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14.  Abstract 


Cognitive  task  analysis  is  defined  as  the  extension  of  traditional  task  analysis  techniques  to 
yield  information  about  the  knowledge,  thought  processes  and  goal  structures  that  underlie 
observable  task  performance.  Cognitive  task  analyses  are  conducted  for  a  wide  variety  of 
puiposes,  including  the  design  of  computer  systems  to  support  human  work,  the  development  of 
training,  and  the  development  of  tests  to  certify  competence.  As  part  of  its  Programme  of 
Work,  NATO  Research  Study  Group  27  on  Cognitive  Task  Analysis  has  undertaken  the  task  of 
reviewing  existing  cognitive  task  analysis  techniques.  The  Group  concludes  that  few  integrated 
methods  exist,  that  little  attention  is  being  paid  to  the  conditions  under  which  methods  are 
appropriate,  and  that  often  it  is  unclear  how  the  products  of  cognitive  task  analysis  should  be 
used. 

RSG.27  has  also  organized  a  workshop  with  experts  in  the  field  of  cognitive  task  analysis.  The 
most  important  issues  that  were  discussed  during  the  workshop  were:  (1)  the  use  of  CTA  in  the 
design  of  new  systems,  (2)  the  question  when  to  use  what  technique,  and  (3)  the  role  of  CTA 
in  system  design. 

RSG.27  emphasizes:  (1)  that  is  important  for  the  CTA  community  to  be  able  to  empirically 
demonstrate  the  added  value  of  a  CTA;  (2)  it  is  critical  for  the  success  of  CTA  to  be  involved 
in  the  design  process  from  the  start  to  finish,  and  to  establish  clear  links  with  methods  that  are 
used  by  other  disciplines,  and  (3)  recommends  that  more  research  effort  be  directed  to  the  issue 
of  the  reliability  of  CTA  techniques. 
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RTO  is  the  single  focus  in  NATO  for  Defence  Research  and  Technology  activities.  Its  mission  is  to  conduct  and  promote 
cooperative  research  and  information  exchange.  The  objective  is  to  support  the  development  and  effective  use  of  national 
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provide  advice  to  NATO  and  national  decision  makers.  The  RTO  performs  its  mission  with  the  support  of  an  extensive 
network  of  national  experts.  It  also  ensures  effective  coordination  with  other  NATO  bodies  involved  in  R&T  activities. 
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military  users  and  other  NATO  activities,  a  small  part  of  the  RTA  staff  is  located  in  NATO  Headquarters  in  Brussels.  The 
Brussels  staff  also  coordinates  RTO’s  cooperation  with  nations  in  Middle  and  Eastern  Europe,  to  which  RTO  attaches 
particular  importance  especially  as  working  together  in  the  field  of  research  is  one  of  the  more  promising  areas  of  initial 
cooperation. 

The  total  spectrum  of  R&T  activities  is  covered  by  7  Panels,  dealing  with: 

•  SAS  Studies,  Analysis  and  Simulation 

•  SCI  Systems  Concepts  and  Integration 

•  SET  Sensors  and  Electronics  Technology 

•  1ST  Information  Systems  Technology 

•  AVT  Applied  Vehicle  Technology 
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Cognitive  Task  Analysis 

(RTO  TR-24) 


Executive  Summary 


DEFINITION  OF  TERMS 

Cognitive  task  analysis  is  the  extension  of  traditional  task  analysis  techniques  to  yield  information 
about  the  knowledge,  thought  processes  and  goal  structures  that  underlie  observable  task  performance. 
In  recent  years,  interest  in  the  cognitive  aspects  of  tasks  has  grown  for  several  reasons.  Modern 
automated  systems  have  created  jobs  for  humans  that  are  conspicuously  cognitive  in  character, 
emphasizing  inference,  diagnosis,  judgment  and  decision-making.  The  term  ‘Cognitive  Task  Analysis’ 
(CTA)  began  to  emerge  in  reports  in  the  early  1980’s  and  it  encapsulates  attempts  to  apply  current 
concepts  in  cognitive  psychology  to  the  analysis  of  complex  tasks.  Whereas  in  the  1950’s  and  60’ s  the 
major  emphasis  was  on  control  tasks  (flying,  steering,  managing  chemical  plant),  CTA  has  been 
primarily  concerned  with  decision-making  tasks  such  as  air  traffic  control  and  military  command  and 
control  (C2). 

FITERATURE  REVIEW 

As  part  of  its  Programme  of  Work,  RSG.27  on  Cognitive  Task  Analysis  has  undertaken  the  task  of 
reviewing  existing  cognitive  task  analysis  techniques  and  computer  tools.  An  analysis  of  the  reviews 
themselves  shows  that  a  very  large  number  of  particular,  rather  limited  methods  are  described  over  and 
over  again.  But  little  is  said  about  how  these  can  be  effectively  orchestrated  into  an  approach  that  will 
yield  a  complete  analysis  of  a  task  or  job.  Little  is  said  about  the  conditions  under  which  an  approach 
or  method  is  appropriate.  The  literature  is  also  very  weak  when  it  comes  to  specifying  the  way  in 
which  the  products  of  task  analysis  should  be  used  in  designing  either  training  or  systems  with  which 
humans  will  interact. 

WORKSHOP 

RSG.27  has  organized  a  workshop  with  experts  in  the  field  of  cognitive  task  analysis.  The  Workshop 
was  held  in  Washington,  D.C.,  USA,  October  30-November  1,  1997.  The  goal  of  the  workshop  was  to 
assess  the  state  of  the  art  by  bringing  together  a  diverse,  yet  representative  sample  of  experts  in  the 
field. 

The  most  important  issues  that  were  discussed  during  the  workshop  were: 

1.  The  use  of  CTA  in  the  design  of  new  systems.  The  difficulty  lies  in  the  fact  that  when  we  try  to 
predict  future  operator  behavior,  we  can  only  rely  on  existing  operator  behavior. 

2.  The  question  when  to  use  what  technique.  It  was  generally  felt  that  a  strict  and  fixed  ordering  of 
techniques  for  each  project  was  unrealistic,  as  the  order  in  which  techniques  are  used  may  differ 
from  project  to  project.  Also,  use  of  a  single  technique  was  generally  felt  to  be  too  restrictive: 
multiple,  coordinated  approaches  to  CTA  are  required. 

3.  The  role  of  CTA  in  system  design.  Most  speakers  agreed  that  CTA  should  not  be  a  self-contained 
activity,  the  results  of  which  are  handed  off  to  system  designers.  However,  this  being  said,  it  is  still 
very  often  the  case  that  there  is  no  integration  between  the  activities  of  cognitive  psychologists  and 
system  designers  (e.g.,  software  engineers). 
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MAJOR  RECOMMENDATIONS 


1 .  It  is  important  for  the  CTA  community  to  be  able  to  empirically  demonstrate  the  added  value  of  a 
CTA.  In  this  way,  the  analyst  goes  beyond  mere  observations  and  submits  his  or  her  ideas  to 
empirical  tests.  Clear  performance  factors  need  to  be  chosen  and  to  be  engineered  toward. 
Examples  of  performance  factors  are  reaction  time,  training  time,  faults  detected,  firepower, 
coverage  of  weapon  systems,  etc. 

2.  It  is  critical  for  the  success  of  CTA  to  be  involved  in  the  design  process  from  the  start  to  finish, 
and  to  establish  clear  links  with  methods  that  are  used  by  other  disciplines. 

3.  Due  to  the  fact  that  customers  are  rarely  willing  to  pay  for  an  extra  CTA,  or  the  same  CTA  carried 
out  by  a  different  analyst,  empirical  tests  of  the  reliability  of  CTA  techniques  are  rare.  More 
research  effort  should  be  devoted  to  this  area. 

MILITARY  IMPLICATIONS 

The  digitization  of  military  forces  increases  the  importance  of  cognitive  work  relative  to  physical 

work.  User-centered  design  is  a  key  success  factor  in  the  introduction  of  computers  into  military 

forces.  Cognitive  task  analysis  techniques  constitute  a  major  part  of  the  user-centered  design  process. 

Therefore,  improving  the  effectiveness  and  efficiency  of  these  techniques  ultimately  leads  to  higher 

user  acceptance  of  digitized  systems  and  increased  combat  power. 

The  military  benefit  of  Cognitive  Task  Analysis  lies  in  the  following  areas: 

(1)  better  match  between  system  functions  and  human  cognitive  capabilities 

(2)  optimization  of  system  performance  and  workload 

(3)  improved  operational  Command  and  Control  team  performance 

(4)  better  understanding  of  abilities  needed  for  the  job 

(5)  enhanced  training  system  design 


1 ’Analyse  des  taches  cognitives 

(RTO  TR-24) 


Synthese 


DEFINITION  DES  TERMES  UTILISES 

L'analyse  des  taches  cognitives  s’inscrit  dans  la  continuity  des  techniques  traditionnelles  d’analyse  des 
taches  destinees  a  fournir  des  informations  concernant  les  connaissances,  le  fonctionnement  de  la 
pensee  et  la  definition  d’objectifs  qui  sous-tendent  1’ execution  observable  des  taches.  Au  cours  des 
dernieres  annees,  de  plus  en  plus  d'interet  a  ete  exprime  pour  les  aspects  cognitifs  des  taches  et  ce  pour 
de  nombreuses  raisons.  Les  systemes  modernes  automatises  ont  cree  des  emplois  qui  sont 
manifestement  cognitifs  du  point  de  vue  de  leur  nature,  de  T accentuation  de  T  inference,  du  diagnostic, 
du  jugement  et  de  la  prise  de  decisions.  Le  terme  “Analyse  des  taches  cognitives”  (CTA)  est  apparu 
pour  la  premiere  fois  dans  certains  rapports  au  debut  des  annees  1980.  II  englobe  les  efforts  qui  ont  ete 
faits  pour  appliquer  les  concepts  actuels  de  la  psychologic  cognitive  a  l’analyse  de  taches  complexes. 
Alors  que  dans  les  annees  1950  et  1960  l’accent  majeur  avait  ete  mis  sur  les  taches  de  controle 
(pilotage,  guidage,  gestion  de  produits  chimiques),  aujourd’hui,  le  CTA  est  principalement  utilise  pour 
des  taches  decisionnelles  telles  que  le  controle  de  la  circulation  aerienne  et  le  C2  militaire. 

ETAT  DES  DOCUMENTS  DISPONIBLES 

Dans  le  cadre  de  son  programme  de  travail,  le  groupe  de  recherche  scientifique  RSG27  sur  l’analyse 
des  taches  cognitives  a  passe  en  revue  les  techniques  existantes  d’analyse  des  taches  cognitives  et  des 
outils  informatiques  associes.  Cette  analyse  montre  qu'il  s’agit  en  fait  de  descriptions  repetees  d’un 
tres  grand  nombre  de  methodes  specifiques  et  quelque  peu  limitees.  Peu  d’ elements  sont  disponibles 
sur  la  maniere  de  faire  la  synthese  de  ces  methodes  pour  aboutir  a  une  analyse  complete  d’une  tache  ou 
d’un  travail.  De  meme,  il  n’y  a  que  tres  peu  d’indications  sur  les  conditions  et  l’adequation  d’une 
approche  ou  d’une  methode  donnee.  Enfin,  ce  passage  en  revue  a  permis  de  constater  que  presque  rien 
ne  portait  sur  la  definition  d’un  cheminement  permettant  d’utiliser  les  resultats  d’analyse  de  taches 
pour  la  conception  soit  de  programmes  d’entrainement,  soit  de  systemes  impliquant  des  interactions 
homme-machine . 

ATELIER 

Le  groupe  RSG27  a  organise  un  atelier  de  specialistes  dans  le  domaine  de  l’analyse  des  taches 
cognitives.  Cet  atelier  s’est  tenu  a  Washington,  D.C.,  aux  Etats-Unis,  du  30  octobre  au  ler  novembre 
1997.  II  a  eu  pour  objectif  de  faire  le  point  de  l’etat  actuel  des  connaissances  en  reunissant  un  groupe 
de  specialistes  d’origines  diver ses  mais  representatifs  dans  ce  domaine. 

Les  questions  les  plus  importantes  qui  ont  ete  discutees  lors  de  1’  atelier  furent  les  suivantes  : 

1 .  La  mise  en  oeuvre  du  CTA  pour  la  conception  de  nouveaux  systemes.  La  difficulty  reside  dans  le 
fait  que  la  seule  base  disponible  pour  la  prevision  du  comportement  des  operateurs  futurs  est  le 
comportement  des  operateurs  actuels. 

2.  La  question  de  savoir  quand  utiliser  une  technique  donnee.  De  l’avis  general,  V  attribution  stricte  et 
immuable  d’une  technique  donnee  a  un  projet  donne  serait  peu  realiste,  car  l’ordre  dans  lequel  les 
differentes  techniques  sont  mises  en  oeuvre  change  de  projet  en  projet.  Parallelement,  l’emploi 
d’une  seule  technique  serait  trap  contraignant.  Au  contraire,  des  approches  differentes  du  CTA 
multiples  et  coordonnees  sont  recommandees. 

3.  Le  role  du  CTA  dans  la  conception  des  systemes.  La  majority  des  intervenants  etaient  de  l’avis  que 
le  CTA  ne  devrait  pas  etre  une  activity  autonome,  dont  les  resultats  seraient  a  distribuer  aux 
concepteurs  de  systemes.  Cependant,  il  est  souvent  constate  qu’aucune  integration  n’existe  entre 


les  activites  des  psychologues  cognitifs  et  les  concepteurs  de  systemes  (par  exemple  les  ingenieurs 
en  logiciel). 

RECOMMANDATIONS  PRINCIPALES 

1.  II  est  important  pour  la  communaute  CTA  de  pouvoir  demontrer  la  valeur  ajoutee  du  CTA.  Ainsi, 
l’analyste  va  au-dela  des  simples  observations  et  soumet  ses  idees  a  des  essais  empiriques.  II  y  a 
lieu  de  definir  clairement  des  facteurs  de  performance  et  de  les  incorporer  dans  le  processus  de 
conception.  Des  exemples  de  facteurs  de  performance  sont  :  les  temps  de  reponse,  les  durees  de 
formation,  les  defauts  detectes,  la  puissance  de  feu,  la  couverture  des  systemes  d’armes,  etc... 

2.  L' acceptation  du  CTA  passe  par  son  implication  a  tous  les  niveaux  du  processus  de  conception, 
ainsi  que  par  la  creation  de  liens  avec  des  methodes  mises  en  oeuvre  dans  d’autres  disciplines. 

3.  Des  essais  empiriques  de  la  fiabilite  des  techniques  du  CTA  sont  rares  en  raison  du  fait  que  les 
clients  sont  peu  enclins  a  payer  des  CTA  supplementaires  ou  payer  le  meme  CTA  realise  par  un 
analyste  different.  II  y  a  lieu  de  redoubler  les  efforts  de  recherche  consacres  a  ce  domaine. 

CONSEQUENCES  MILITARIES 

La  numerisation  des  forces  militaires  met  en  evidence  T  importance  des  travaux  cognitifs  sur  le  travail 
physique.  La  conception  orientee  utilisateur  est  un  facteur  cle  dans  1' informatisation  des  forces 
militaires.  Les  techniques  d’ analyse  des  taches  cognitives  constituent  une  partie  importante  du 
processus  de  la  conception  orientee  utilisateur.  Par  consequent,  toute  amelioration  de  l'efficacite  de  ces 
techniques  conduira  a  une  meilleure  acceptation  par  T  utilisateur  des  systemes  numerises  et  a  une  plus 
grande  puissance  de  combat. 

Les  avantages  militaires  de  T  analyse  des  taches  cognitives  peuvent  etre  resumes  comme  suit  : 

(1)  meilleure  adequation  entre  les  fonctions  systeme  et  les  capacites  cognitives  humaines 

(2)  optimisation  des  performances  systeme  et  de  la  charge  de  travail 

(3)  meilleure  performance  de  l’equipe  operationnelle  de  commandement  et  controle 

(4)  meilleure  comprehension  des  capacites  demandees  pour  une  tache  donnee 

(5)  meilleure  conception  des  systemes  d’entrainement 


Click  inside  the  blue  boxes  to  view  the  corresponding  section 
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Preface 


For  almost  a  century,  psychologists  have  been  interested  in  matching  tasks  with  human  capabilities.  The  ultimate 
goal  of  this  endeavour  was,  and  still  is,  to  make  work  safe,  productive,  and  healthy.  Matching  tasks  with  human 
capabilities  requires  a  deep  understanding  both  of  the  task  domain  and  the  human  practitioner.  Over  the  past  100 
years,  various  techniques  have  been  at  the  psychologist’s  disposal,  depending  on  the  nature  of  work.  Before 
World  War  II,  psychologists  mainly  analysed  manual  work  with  the  aim  of  improving  the  efficiency  of  every 
step  in  the  work  process.  This  changed  in  the  1950’ s  when  the  number  of  white-collar  workers  started  to  exceed 
the  number  of  bine-collar  workers.  New  techniques  were  required  for  analysing  mental  rather  than  manual  work. 
Cognitive  task  analysis  is  a  broad  area  consisting  of  tools  and  techniques  for  describing  the  knowledge  and 
strategies  required  for  task  performance.  Cognitive  task  analysis  has  implications  for  the  development  of  expert 
systems,  training  and  instructional  design,  expert  decision-making  and  policy-making.  It  has  been  applied  in  a 
wide  range  of  settings,  with  different  purposes,  for  instance:  specifying  user  requirements  in  system  design  or 
specifying  training  requirements  in  training  needs  analysis. 

Several  historical  developments  have  contributed  to  what  we  now  call  ‘cognitive  task  analysis’.  First  of  all,  in 
the  late  1950’s,  it  became  clear  that  computers  were  not  just  number-crunchers,  but  rather  general-purpose 
symbol  manipulation  machines.  Human  mental  work,  in  particular  problem  solving,  could  be  considered  a  form 
of  symbol  manipulation.  Hence,  the  argument  went  on;  human  mental  work  could  be  computationally  modelled. 
The  technique  used  to  do  such  computational  modelling  was  ‘protocol  analysis’,  followed  by  the  implementation 
in  the  form  of  production  rules.  This  school  of  thought,  prevalent  at  Carnegie  Mellon  University,  culminated  in 
the  seminal  work  by  Newell  and  Simon  on  “Human  Problem  Solving”  (1972),  and  continues  today  in  the  work 
of  Anderson  and  colleagues  (1983;  1993;  Anderson  &  Lebiere,  1998).  Although  the  computational  modelling 
approach  is  first  and  foremost  interested  in  understanding  the  architecture  of  cognition  for  its  own  sake,  practical 
spin-offs  have  been  intelligent  tutoring  systems  and  expert  systems. 

A  second  development,  in  the  1960’s,  was  the  growing  interest  of  psychologists  in  supervisory  control  tasks  in 
the  military  and  in  civilian  industry.  Particularly  in  Great  Britain,  interest  in  training  issues  in  process  control  led 
to  a  general  approach  to  study  tasks  and  task  demands,  called  “Hierarchical  Task  Analysis”  (HTA)  (Annett  & 
Duncan,  1967).  On  the  basis  of  an  HTA,  individual  techniques  can  be  chosen  to  solve  particular  problems,  such 
as  a  lack  of  knowledge  by  an  operator. 

A  third  development  was  a  number  of  accidents  in  nuclear  and  chemical  industries  (e.g.,  the  Three  Mile  Island 
nuclear  power  plant  accident),  which  spurred  research  into  how  to  design  tools  for  operators  that  would  prevent 
such  accidents.  This  led  to  the  “cognitive  systems  engineering”  approach  in  the  early  1980’s  (e.g.,  Hollnagel  & 
Woods,  1983).  In  this  approach,  techniques  for  capturing  user  requirements  are  very  important. 

A  fourth  development  was  a  growth  in  knowledge  in  the  basis  for  expertise  (Glaser,  1984),  and,  in  parallel,  the 
commercialisation  of  expert  systems,  starting  in  the  early  1980’s  (Hayes-Roth,  Waterman,  &  Lenat,  1983).  It 
soon  became  commonplace  to  refer  to  the  capturing  of  expert  knowledge  as  the  “knowledge  elicitation 
bottleneck”,  as  this  seemed  to  be  the  most  time-consuming  activity,  relative  to  implementing  the  knowledge  in 
the  expert  system.  Numerous  books  and  articles  appeared,  describing  literally  dozens  of  techniques  with  which 
knowledge  could  be  captured.  An  early  attempt  to  combine  the  research  on  expert-novice  differences  with 
knowledge  elicitation  techniques,  can  be  found  in  Schraagen  (1986). 

These  multiples,  parallel  and  independent  origins  all  contributed  to  what  has  since  the  beginning  of  the  1980’s 
been  referred  to  as  “cognitive  task  analysis”.  Cognitive  task  analysis  is  an  activity  that  is  not  carried  out  for  its 
own  sake;  rather,  its  primary  aim  is  to  dissect  “mental  work”  into  more  manageable  constructs  that  shed  light  on 
a  particular  problem.  The  choice  of  constructs  is  determined  by  the  techniques  chosen  by  the  analyst,  and  these 
are  largely  determined  by  one’s  theoretical  inclinations. 

The  increasing  importance  of  cognitive  task  analysis  for  military  system  development  was  recognised  by 
NATO’s  Panel  8  (Defence  Applications  of  Human  and  Bio-  Medical  Sciences)  Defence  Research  Group  in  1994, 
when  an  Exploratory  Group  on  Cognitive  Task  Analysis  was  established.  The  Group  was  chaired  by  Dr.  J.M.C. 
Schraagen  (TNO  Human  Factors  Research  Institute,  The  Netherlands),  and  met  once  in  Soesterberg  to  draft  the 
Terms  of  Reference  and  a  Programme  of  Work  for  a  subsequent  Research  Study  Group  (RSG).  This  RSG  was 
formally  established  in  the  fall  of  1995  as  RSG.27  on  Cognitive  Task  Analysis.  The  participating  countries  were: 
The  Netherlands  (lead  nation).  United  Kingdom,  France,  and  the  United  States  of  America.  In  1997,  Germany 
joined  the  RSG.  RSG.27  met  six  times  in  three  years,  starting  in  January,  1996  (in  chronological  order: 


viii 


Soesterberg,  Montreuil-Juigne,  London,  Washington,  Soesterberg,  and  Versailles).  Its  main  activities  consisted  of 
(a)  writing  a  state-of-the-art  review  on  cognitive  task  analysis  techniques,  (b)  organising  a  workshop  with 
experts  in  the  field  in  1997  in  Washington,  DC,  and  (c)  identifying  new  developments  and  issues  for  further 
research.  The  papers  presented  at  the  workshop  were  edited  extensively  by  Schraagen,  Chipman,  and  Shalin,  and 
will  be  published  by  Lawrence  Erlbaum  Associates  under  the  title  “Cognitive  Task  Analysis”. 

The  present  Technical  Report  is  the  final  deliverable  of  RSG.27  to  NATO  RTO.  It  consists  of  (a)  a  general 
introduction  to  the  field  of  cognitive  task  analysis  (also  to  be  published  as  chapter  1  in  Schraagen,  Chipman,  & 
Shalin,  in  press),  (b)  a  state-of-the-art  review  of  cognitive  task  analysis  techniques,  and  (c)  a  report  on  the 
workshop  with  experts  in  the  field. 
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SUMMARY 

This  introductory  chapter  will  define  cognitive  task  analysis  and  give  the  authors’  view  of  the  state  of  the  art.  Cognitive 
task  analysis  is  defined  as  the  extension  of  traditional  task  analysis  techniques  to  yield  information  about  the  knowledge, 
thought  processes  and  goal  structures  that  underlie  observable  task  performance.  Cognitive  task  analyses  are  conducted  for  a 
wide  variety  of  purposes,  including  the  design  of  computer  systems  to  support  human  work,  the  development  of  training, 
and  the  development  of  tests  to  certify'  competence.  The  introductory'  chapter  will  draw  upon  a  recent  review  of  the  literature 
on  cognitive  task  analysis,  winch  was  conducted  by  RSG.27.  From  that  review,  an  image  of  the  prototvpic  ideal  case  of  a 
cognitive  task  analysis  process  emerges.  The  subsequent  phases  in  the  process  are  discussed.  The  introduction  ends  with  a 
discussion  of  further  research  needed  in  tire  field  of  cognitive  task  analysis.  Some  important  issues  are  the  use  of  cognitive 
task  analysis  in  tire  design  of  new  systems,  tire  development  of  systematic  approaches  that  effectively  integrate  tire  many 
available  techniques,  tire  appropriate  selection  of  approaches  for  a  given  problem  and  applied  purpose,  and  the  application  of 
cognitive  task  analysis  to  team  tasks. 

INTRODUCTION 

Modem  work,  with  its  increasing  reliance  on  automation  to  support  human  action,  is  focusing  attention  on  the  cognitive 
aspects  of  work  that  are  not  accessible  to  direct  observation.  For  example,  it  is  obvious  that  the  physical  acts  of  button 
pushing  that  occur  in  tire  command  center  of  a  modem  ship  are  of  less  intrinsic  importance  than  tire  mental  decision 
processes  executed  via  those  actions.  The  mental  processes  organize  and  give  meaning  to  the  observable  physical 
actions.  Attempts  to  analyze  a  task  like  air  traffic  control  with  traditional  behavioral  task  analysis  techniques  made  the 
shortcomings  of  those  techniques  strikingly  clear  (Means,  1993).  Starting  in  the  1960s,  the  cognitive  revolution  in 
academic  psychology  has  both  increased  our  awareness  of  the  extensive  cognitive  activity'  underlying  even  apparently 
simple  tasks  and  provided  research  techniques  and  theories  for  characterizing  covert  cognition.  Hence,  the  term 
cognitive  task  analysis  is  coming  into  use  to  describe  a  new  branch  of  applied  psychology'.  The  relative  newness  of  tins 

1  This  chapter  is  an  abbreviated  version  of  Chapter  1  in  J.M.C.  Schraagen,  S.E.  Chipman,  and  V.L.  Shalin  (Eds.), 
Cognitive  task  analysis  (in  press).  Mahwah,  NJ:  Lawrence  Erlbaum  Associates.  Published  by  permission  of  Law  rence 
Erlbaum  Associates,  Inc. 
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enterprise  is  evidenced  by  the  fact  that,  as  of  this  writing,  a  search  of  the  entire  Psychlnfo  database  with  the  term 
yielded  only  28  items,  some  irrelevant,  and  a  search  in  the  Science  Citation  Index  yielded  30  items.  The  high  current 
interest  in  cognitive  task  analysis  is  evidenced  by  recent  literature  review  efforts  undertaken  by  a  British  aerospace 
company  (confidential)  and  by  the  French  military  (Doireau,  Grau,  &  Poisson,  1996)  as  well  as  the  NATO  Study  Group 
effort  reported  here. 


Cognitive  task  analysis  is  the  extension  of  traditional  task  analysis  techniques  to  yield  information  about  the  knowledge, 
thought  processes,  and  goal  structures  that  underlie  observable  task  performance.  Some  would  confine  the  term 
exclusively  to  the  methods  that  focus  on  the  cognitive  aspects  of  tasks,  but  this  seems  counterproductive.  Overt 
observable  behavior  and  the  covert  cognitive  functions  behind  it  form  an  integrated  whole.  Artificially  separating  and 
focusing  on  the  cognitive  alone  is  likely  to  produce  information  that  is  not  very  useful  in  understanding,  aiding,  or 
training  job  performance.  The  tension  between  traditional  behavioral  task  analysis  techniques  and  newer  cognitive  task 
analysis  is  largely  a  U.S.  phenomenon.  Elsewhere,  behaviorism  never  took  hold  as  it  did  in  the  U.S.,  where  military 
regulations  governing  training  development  have  forbidden  talk  of  processes  that  go  on  inside  the  head  almost  until  the 
present  day.  Annett,  Duncan,  Stammers,  and  Gray’s  (1971)  hierarchical  task  analyses,  for  example,  often  segued 
smoothly  from  the  domain  of  observable  behavior  to  the  internal  world  of  perception  and  cognition.  The  changing 
nature  of  work,  however,  is  universal  throughout  the  developed  world.  Even  those  who  did  not  eschew  analysis  of  the 
cognitive  aspects  of  work  now  need  more  powerful  tools  and  techniques  to  address  the  large  role  of  cognition  in  modem 
work. 

Analyses  of  jobs  and  their  component  tasks  may  be  undertaken  for  a  wide  variety  of  purposes,  including  the  design  of 
computer  systems  to  support  human  work,  the  development  of  training,  or  the  development  of  tests  to  certify  job 
competence.  An  emerging  frontier  of  modem  task  analysis  is  the  analysis  of  entire  working  teams  activities.  Thrs  is 
done  for  purposes  such  as  the  allocation  of  responsibilities  to  individual  humans  and  cooperating  computer  systems, 
often  with  the  goal  of  reducing  the  number  of  humans  who  must  be  employed  to  accomplish  the  team  function.  Given 
the  purposes  and  constraints  of  particular  projects,  several  (cognitive)  task  analysis  approaches  merit  consideration. 
Savvy  customers  and  practitioners  of  cognitive  task  analysis  must  know  that  one  approach  will  not  fit  all  circumstances. 
On  the  other  hand,  a  thoroughgoing  cognitive  task  analysis  may  repay  the  substantial  investment  required  by  proving 
applicable  to  purposes  beyond  the  original  intent.  For  example,  Zachary,  Ryder,  and  Hicinbothom  (in  press)  analyzed 
the  tasks  of  the  AEGIS  antiair  warfare  team  in  order  to  build  an  artificially  intelligent  training  system,  but  these  same 
analyses  are  being  used  to  guide  the  design  of  advanced  work  stations  and  new  teams  with  fewer  members. 

This  report  is  the  ultimate  product  of  a  NATO  study  group  aiming  to  capture  the  state  of  the  art  of  cognitive  task 
analysis.  The  intent  is  to  advance  it  toward  a  more  routine  engineering  discipline — one  that  could  be  applied  reliably  by 
practitioners  not  necessarily  educated  at  the  doctoral  level  in  cognitive  psychology  or  cognitive  science.  To  that  end, 
two  major  activities  were  undertaken.  One  was  a  review  of  the  state  of  the  art  of  cognitive  task  analysis,  focusing  on 
recent  articles  and  chapters  claiming  to  review  cognitive  task  analysis  techniques.  This  effort  produced  a  bibliographic 
resource  appearing  as  a  chapter  in  this  report.  We  hope  that  this  chapter  gives  sufficient  information  to  help  students 
and  other  readers  decide  which  of  these  earlier  contributions  to  the  field  they  should  read  for  their  particular  purposes. 
The  second  major  activity  of  the  NATO  study  group  was  an  international  workshop  intended  to  provide  an  up-to-date 
snapshot  of  cognitive  task  analyses,  emphasizing  new  developments.  Invitations  were  extended  to  known  important 
contributors  to  the  field.  The  opportunity  to  participate  was  also  advertised  widely  through  electronic  mailing  lists,  in 
order  to  capture  new  developments  and  ongoing  projects  that  might  not  be  known  to  the  study  group  members 
organizing  the  workshop.  This  report  is  largely  the  product  of  that  workshop,  sharing  its  insights  into  the  state  of  the  art 
of  this  new  field.  This  introduction  provides  an  overview  of  these  two  activities.  First,  we  sketch  the  prototype  cognitive 
task  analysis  based  on  results  from  the  NATO  study  group. 

THE  PROTOTYPIC  COGNITIVE  TASK  ANALYSIS  PROCESS  AS  SEEN  IN  PRIOR  LITERATURE 

Ironically,  the  cognitive  analysis  of  tasks  is  itself  a  field  of  expertise  like  those  it  attempts  to  describe.  Reviewing  recent 
discussions  of  cognitive  task  analysis  reveals  that  the  explicitly  stated  state  of  the  art  is  lacking  specification  of  just  those 
kinds  of  knowledge  most  characteristic  of  expertise.  A  large  number  of  particular,  limited  methods  are  described  repeatedly. 
However,  little  is  said  about  how  these  can  be  effectively  orchestrated  into  an  approach  that  will  yield  a  complete  analysis  of 
a  task  or  job.  Little  is  said  about  the  conditions  under  which  an  approach  or  method  is  appropriate.  Clearly,  the  relevant 
conditions  that  need  to  be  considered  include  at  least  the  type  of  task  being  analyzed,  the  purpose  for  which  the  analysis  is 
being  done  (HCI;  human-computer  interaction  design,  training,  testing,  expert  system  development),  and  the  resources 
available  for  the  analysis,  particularly  the  type  of  personnel  available  to  do  the  analysis  (cognitive  scientists,  cognitive 
psychologists,  educational  specialists,  subject-matter  experts).  The  literature  is  also  weak  in  specifying  the  way  in  which  the 
products  of  task  analysis  should  be  used  in  designing  either  training  or  systems  with  which  humans  will  interact.  The  prior 
literature  on  cognitive  task  analysis  is  also  limited  by  a  focus  on  the  tasks  of  individuals,  almost  exclusively  existing  tasks 
for  which  there  are  existing  task  experts. 
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Nevertheless,  the  literature  review  effort  did,  within  these  limits,  provide  the  image  of  a  prototypic  ideal  case  of  the 
cognitive  task  analysis  process,  as  it  might  be  when  unhampered  by  resource  limitations.  What  emerges  as  the  ideal  case, 
assuming  that  resource  limitations  are  not  a  problem?  Although  the  answer  to  this  question  may  vary  somewhat,  depending 
the  purpose  for  which  the  analysis  is  being  done,  we  set  aside  that  consideration  for  a  while  or  assume  that  the  purpose  is 
training  and  associated  proficiency  measurement.  Several  of  the  articles  we  reviewed  are  strong  in  their  presentation  of  an 
inclusive  recommended  approach  to  cognitive  task  analysis  (e.g.,  Hall,  Gott,  &  Pokomy,  1995,  Hoffman,  Shadbolt,  Burton, 
&  Klein,  1995;  Means,  1993;  DuBois  &  Shalin,  1995). 

Preliminary  Phase 

One  should  begin  a  cognitive  task  analysis  with  a  study  of  the  job  or  jobs  involved  to  determine  what  tasks  merit  the  detailed 
attention  of  a  cognitive  task  analysis.  Standard  approaches  from  personnel  psychology  are  appropriate  for  this  phase  of  the 
effort,  using  unstructured  interviews  and/or  questionnaires  to  determine  the  importance,  typicality,  and  frequency  of  tasks 
within  job  performance.  Hall  et.  al.,  (1995)  discussed  this  preliminary  phase,  as  did  DuBois  and  Shalin  (1995)  with 
somewhat  more  methodological  detail.  DuBois  and  Shalin  also  pointed  out  the  importance  of  focusing  on  the  tasks  or 
problems  within  general  tasks  that  discriminate  more  expert  performance  from  routine  performance,  even  though  these  may 
not  be  high-frequency  events.  Klein  Associates'  approach  seems  to  embody  the  same  view,  with  an  emphasis  on  gathering 
data  about  past  critical  incidents  in  experts'  experience. 

Depending  on  the  availability  of  written  materials  about  the  job  or  task,  such  as  existing  training  materials,  the  first  step  for 
those  responsible  for  the  analysis  probably  should  be  to  read  those  materials  to  gain  a  general  familiarity  with  the  job  or  task 
and  a  knowledge  of  the  specialized  vocabulary  (this  is  referred  to  as  bootstrapping  by  Hoffinan  et  al.,  (1995),  and  table-top 
analysis  by  Flach  (in  press).  The  major  alternative  is  to  begin  with  informal,  unstructured  interviews  with  persons  who  have 
been  identified  as  experts.  In  the  ideal  case,  the  task  analysis  becomes  a  team  effort  among  one  or  more  experts  in  cognitive 
task  analysis  and  several  subject-matter  experts.  Of  course,  it  is  important  to  obtain  the  time,  effort,  and  cooperation  of 
experts  who  are  in  fact  expert.  Hall  et.  al.  (1995)  discussed  the  issue  of  the  scarcity  of  true  experts  and  the  selection  of 
appropriate  experts  in  moderate  detail.  Hoffinan  et.  al.(1995)  were  also  concerned  with  the  gradations  of  expertise. 
Articulate  experts  with  recent  experience  in  both  performing  and  teaching  the  skill  are  particularly  useful.  For  example,  the 
MYCIN  (Buchanan  &  Shortliffe,  1984  expert  was  reknowned  for  his  ability  to  teach  medical  diagnosis.  It  is  also  true  that 
not  just  anyone  is  suitable  for  acting  as  a  cognitive  task  analyst— not  even  just  anyone  who  is  educated  in  cognitive 
psychology  and  cognitive  science.  Analysts  must  have  the  social  skills  to  establish  rapport  with  the  subject-matter  experts 
(SMEs),  sometimes  across  the  barriers  of  different  social  cultural  and  economic  backgrounds.  If  doing  unstructured  or  even 
structured  interviews,  they  must  be  verbally  adept  to  adapt  to  the  changing  circumstances  of  the  interview.  They  must  be 
intelligent,  quick  learners  because  they  have  to  learn  a  great  deal  about  the  task  to  analyze  it  effectively.  Hoffinan  et  al. 
(1995)  and  Crandall,  Klein,  Militello,  and  Wolf  (1994)  discussed  some  of  these  issues  about  the  requirements  for  cognitive 
task  analysts.  Forsythe  (1993)  also  appears  to  be  a  reference  of  interest  on  these  points.  There  is  also  a  good  deal  of  literature 
from  the  expert  systems  community  dealing  with  the  practicalities  of  interviewing  and  with  requirements  that  both  the 
knowledge  engineer  and  the  expert  must  meet  (e.g.,  Firlej  &  Hellens,  1991;  McGraw  &  Harbison  Briggs,  1989;  Meyer  & 
Booker,  1991;  Waterman,  1986). 

Identifying  Knowledge  Representations 

A  major  goal  for  the  initial  unstructured  interviews  with  the  SMEs  should  be  to  identify  the  abstract  nature  of  the  knowledge 
involved  in  the  task,  that  is,  the  type  of  knowledge  representations  that  need  to  be  used.  This  can  order  die  rest  of  the  task 
analytic  effort.  This  point  is  not  explicit  in  the  literature,  but  the  more  impressive,  convincing  approaches  are  organized 
around  a  knowledge  representation  or  set  of  knowledge  representations  appropriate  for  the  job  or  task.  For  example,  DuBois 
and  Shalin  (1995;  in  press)  use  a  goal-method  graph  annotated  with  additional  information  about  the  basis  for  method 
selection  and  the  explanation  of  the  rationale  or  principles  behind  the  method.  Less  explicitly,  the  PARI  method  (Hall  et  al., 
1995)  gathers  essentially  the  same  information  supplemented  by  information  about  the  experts'  mental  organization  of 
device  structure  and  function.  Crandall  et  al  (1994)  advocated  collecting  mental  models  of  the  task  and  the  team  context  of 
work,  as  well  as  of  the  equipment.  For  eliciting  knowledge  about  how  a  device  or  system  works,  Williams  and  Kotnur 
(1993)  described  Miyake's  (1986)  constructive  interaction.  Benysh,  Koubek,  and  Calvez  (1993)  proposed  a  knowledge 
representation  that  combines  procedural  information  with  conceptual  information.  Similarly,  in  ongoing  work,  Williams, 
Hultman,  and  Graesser  (1998)  have  collaborated  on  ways  to  combine  the  representations  of  declarative  and  procedural 
knowledge.  Semantic  networks  are  probably  over-represented  in  reviews  of  knowledge  acquisition  methods  relative  to  their 
actual  utility.  Although  measures  of  conceptual  relatedness  or  organization  are  sensitive  to  growth  in  expertise,  they  may 
actually  be  derived  from  more  complex  knowledge  organizations  in  the  experts'  minds,  such  as  those  mentioned  earlier  that 
integrate  procedural  and  declarative  knowledge.  For  example,  it  might  be  a  mistake  to  attempt  to  directly  train  the 
conceptual  organizations  one  deduces  from  studies  of  experts.  However,  semantic  networking  or  clustering  techniques  have 
been  successfully  used  to  structure  more  effective  computer  interfaces  (Patel,  Drury,  &  Shalin,  1998;  Roske-Hofstrand  & 
Paap,  1986;  Vora,  Helander,  &  Shalin,  1994).  As  we  gain  experience  with  cognitive  task  analysis,  it  may  become  possible  to 
define  a  taxonomy  of  tasks  that,  in  effect,  would  classify  tasks  into  types  for  which  the  same  abstract  knowledge 
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representations  and  the  same  associated  knowledge-elicitation  methods  are  appropriate.  However,  we  should  always  keep  in 
mind  the  possibility  that  the  particular  task  of  concern  may  involve  some  type  of  knowledge  not  in  the  stereotype  for  its 
assigned  position  in  the  classification  scheme. 

Knowledge-  Elicitation  Techniques 

Having  identified  the  general  framework  for  the  knowledge  that  has  to  be  obtained,  the  analysts  can  then  proceed  to  employ 
the  knowledge-elicitation  techniques  or  methods  discussed  in  the  articles  reviewed.  Structured  interviews  can  be  used  to 
obtain  information — an  approach  that  is  well  discussed  in  Hoffman  et  al.  (1995),  Randel,  Pugh,  and  Reed  (1996),  and 
Crandall  et  al.  (1994).  The  extreme  of  the  structured  interview  is  the  computer-aided  knowledge-elicitation  approach, 
discussed  in  reviews  by  Williams  and  Kotnour  (1993)  and  Cooke  (1994)  and  exemplified  by  Shute's  (in  press)  DNA 
cognitive  task  analysis  software  and  Williams'  (in  press)  CAT  and  CAT-Human-Computer  Interaction  tools.  The  latter 
structure  and  support  a  generalized  version  of  a  GOMS-style  analysis,  generating  much  the  same  sort  of  goal-method 
representation  recommended  by  DuBois  and  Shalin.  Of  course  these  interviews  and  other  methods  must  be  focused  on  an 
appropriate  representative  set  of  problems  or  cases  previously  identified,  as  alluded  to  earlier.  The  PARI  method  (Hall  et.  al., 
1995)  features  the  participation  of  SMEs  to  develop  appropriate  problems.  The  importance  of  an  appropriate  sample  of 
problems  or  tasks  is  perhaps  most  obvious  for  the  simple  case  of  the  use  of  GOMS  analysis  to  evaluate  alternative 
keyboard/display  designs.  The  basis  for  evaluation  would  be  differences  in  the  predicted  execution  times — for  a 
representative  sample  of  tasks.  In  training  development,  the  issue  is  providing  adequate  coverage  of  essential  knowledge  and 
skills. 

Structured  interviews  and  their  extensions  into  computer-aided  methods  used  by  SMEs  assume  that  the  experts  have 
direct  conscious  access  to  their  relevant  knowledge  and  skill.  However,  research  on  expertise  has  shown  that  this  cannot 
be  assumed.  Much  goes  on  below  the  level  of  conscious  awareness,  especially  in  skills  that  are  exercised  under  time 
pressure  or  have  significant  perceptual  and/or  motor  aspects.  (For  this  reason,  although  the  true  ideal  case  might  seem  to 
be  having  the  cognitive  analyst  and  SME  combined  in  one  person,  as  has  been  true  for  many  of  the  successful 
intelligent  tutoring  projects,  it  is  never  safe  to  assume  that  experts  can  directly  explicate  task  knowledge.)  Often  one 
may  simply  extract  the  expert's  naive  psychological  theory  of  how  the  task  is  performed — a  theory  that  will  not  stand  up 
to  empirical  investigation.  (Obviously  even  expert  cognitive  psychologists  often  propose  theories  of  task  performance 
that  do  not  stand  up  to  empirical  investigation).  For  this  reason,  the  use  of  process-tracing  methods  (cf.  Cooke,  1994)  is 
recommended.  Observation  of  expert  performance,  which  may  be  videorecorded  and  carefully  and  elaborately  coded, 
belongs  in  this  category.  Observation  tends  to  contribute  primarily  to  an  analysis  of  the  overt,  observable  aspects  of  the 
task.  However,  if  the  task  involves  communication  among  team  members,  cognitive  aspects  of  the  task  may  be 
revealed.  Observation  of  apprenticeship  training  is  likely  to  reveal  such  information,  and  there  is  a  variation  of  this  in 
which  the  expert  is  asked  to  coach  an  analyst  in  task  performance.  Most  conspicuous  among  the  cognitive  process 
tracing  methods  is  the  collection  of  verbal  think  aloud  protocols  while  the  SMEs  perform  a  representative  set  of  task 
problems.  (The  PARI  method  involves  the  use  of  SMEs  to  simulate  problems — fault  effects,  test  results  etc. — for  other 
experts  attempting  unfamiliar  problems.)  To  increase  the  information  yield  for  practical  applications,  protocols  may  be 
supplemented  with  probe  questions  or  retrospective  review  of  videotapes  with  probe  questions  (cf.  Dubois  &  Shalin,  in 
press;  Zachary,  Ryder,  &  Hicinbothom,  in  press).  There  is  also  a  variant  approach  called  interruption  analysis  that 
several  reviews  mention.  The  goal  of  these  methods  is  to  bring  out  relevant  knowledge  in  the  context  of  use.  Of  course 
Verbal  protocol  methods  and  other  verbal  methods  are  relatively  ineffective  in  getting  at  nonverbal  types  of  knowledge, 
although  one  may  discover  that  a  surprising  amount  of  specialized  perceptual  vocabulary  has  been  developed  for 
purposes  of  training  and  teamwork.  Experimental  psychology  has  provided  other  process  tracing  methods,  such  as  the 
collection  of  eye  fixations,  that  may  prove  useful  in  getting  at  aspects  of  task  performance  not  represented  verbally. 

If  a  semantic  network  of  concepts  seems  to  be  an  appropriate  part  of  the  representation  of  task  knowledge,  a  large  number  of 
knowledge-elicitation  techniques  are  available  for  this  purpose.  Olson  and  Biolsi  (1991)  provide  die  best  discussion  of 
these — the  associated  data  analysis  methods,  and  the  relations  among  them — along  with  a  useful  diagram  of  those 
relationships.  These  methods  generally  can  be  used  with  only  a  small  set  of  concepts,  a  serious  limitation  given  the  large 
number  of  concepts  actually  involved  in  any  area  of  expertise.  Cooke  (1994)  and  Benyah,  Koubek,  and  Calvez  (1993)  are 
the  only  authors  who  give  much  attention  to  the  problem  of  selecting  the  set  of  concepts  to  be  used  with  these  methods. 

Choosing  target  knowledge  representations  provides  guidance  to  the  overall  cognitive  task  analysis  effort.  A  wide  variety  of 
techniques  from  experimental  psychology,  including  novel  ones  invented  for  the  purpose,  might  be  used  to  fill  in  the 
information  needed  in  one's  chosen  knowledge  representations.  Hoffman  et  al.  (1995)  referred  to  this  deliberate  modification 
of  the  familiar  task  as  the  use  of  contrived  techniques.  Although  contrived  techniques  can  make  the  expert  feel 
uncomfortable  and  may  sometimes  give  misleading  results,  Hoffman  et  al.  noted  that  they  can  be  informative  and  tend  to  be 
more  efficient  in  yielding  information.  Once  they  have  been  developed,  it  is  usual  to  present  the  knowledge  representations 
to  the  SMEs  to  review  for  reasonableness  and  for  comments.  Of  course  if  the  analysis  becomes  the  basis  of  an  expert  system 
or  expert  cognitive  model,  one  may  also  be  able  to  evaluate  its  performance  for  its  resemblance  to  the  performance  of  human 
experts. 
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Using  CTA  Products 

Once  you  have  done  a  cognitive  task  analysis,  how  do  you  make  use  of  the  products  in  your  application?  This  is  a  definite 
weak  point  in  the  literature.  Using  the  output  of  PARI  analyses  to  develop  training  systems  has  proved  problematic  for  the 
U.S.  Air  Force.  If  one  has  a  detailed  goal-method  analysis  down  to  the  production  system  level,  an  analysis  of  the  type  that 
Williams'  CAT  tool  is  designed  to  support,  this  can  probably  be  converted  into  a  running  computational  model  that  can 
function  as  the  student  model  of  an  intelligent  tutoring  system.  John  R.  Anderson  (personal  communication)  has  claimed 
that,  once  such  a  cognitive  model  has  been  developed,  the  actual  development  of  a  tutoring  system  is  quite  trivial,  given  the 
tools  available  in  his  lab.  Of  course  the  process  described  here  would  also  have  yielded  a  set  of  problems  that  could  be  used 
in  the  tutoring  system.  DuBois,  Shalin,  and  their  associates  have  developed  a  method  for  using  task  analysis  results  to 
generate  test  items.  An  elaboration  of  the  same  method  applies  to  the  identification  of  information  requirements  for  interface 
design  (Shalin  &  Geddes,  1994,  1998;  Shalin,  Geddes,  Mikesell,  &  Ramamurthy,  1993).  The  ACTA  CD-ROM  (1997) 
developed  by  Klein  Associates  for  training  cognitive  task  analysis  methods  contains  useful  suggestions  for  the  ways  in 
which  the  information  developed  can  be  used  to  improve  training  (Mitillo  &  Hutton,  1998;  Mitillo  et  al.,  1997).  From  the 
work  of  Schvaneveldt  (1990)  and  his  associates,  it  appears  that  any  of  the  techniques  for  analyzing  concept  organization  can 
be  readily  used  to  provide  crude  assessments  of  the  development  of  expertise  by  measuring  the  resemblance  of  learners' 
conceptual  organization  to  that  of  experts.  In  Schraagen,  Chipman,  and  Shalin  (in  press),  the  chapters  by  Boy  (chap.  18), 
Johnson,  Johnson,  and  Hamilton  (chap.  13);  Neerincx,  van  Doome,  &  Ruijsendaal  (chap.  20);  Ormerod  (chap.  12);  Paris, 
Balbo,  and  Ozkan  (chap.  16);  and  Potter,  Roth,  Woods,  and  Elm  (chap.  21)  attempt  to  bridge  the  gap  between  cognitive  task 
analysis  and  system  design  using  task  models.  Task  models  are  formal  ways  of  describing  standard  task  elements.  They 
serve  to  structure  knowledge  acquisition  and  bridge  the  gap  between  cognitive  task  analysts  and  software  engineers.  As  in 
the  case  of  training,  applications  of  cognitive  task  analysis  to  human-system  interaction  began  by  analyzing  the  interactions 
of  experienced  users  with  existing  systems,  often  to  identify  problematic  features  of  existing  interfaces.  These  methods 
could  be  extended  rather  readily  to  evaluate  detailed  designs  for  new  interfaces  or  to  compare  multiple  designs  for  a  future 
interface,  as  was  done  with  the  GOMS  engineering  model  approach  developed  by  Card,  Moran,  and  Newell  (1983).  The 
GOMS  approach  (analysis  into  Goals,  Operators,  Methods,  and  Selection  criteria  for  methods)  (John  and  Kieras,  1994  is 
conceptually  similar  to  the  tradition  of  hierarchical  task  analysis  dominant  in  the  United  Kingdom. 
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SUMMARY 

Cognitive  task  analysis  is  the  extension  of  traditional  task  analysis  techniques  to  yield  information  about  the  knowledge, 
thought  processes  and  goal  structures  that  underlie  observable  task  performance.  Interest  in  the  cognitive  aspects  of 
behavior  has  grown  since  the  mid  1980's  because  of  the  growing  use  of  automated  systems  in  task  performance.  In  the 
armed  forces,  this  is  denoted  with  the  term  "digitization  of  the  battlefield".  Cognitive  processes  such  as  diagnosis, 
judgment,  planning  and  decision  making  have  come  to  the  fore  as  a  result  of  this  development. 

As  part  of  its  Programme  of  Work,  NATO  Research  Study  Group  27  on  Cognitive  Task  Analysis  has  undertaken  the 
task  of  reviewing  existing  cognitive  task  analysis  techniques.  The  Group  concludes  that  few  integrated  methods  exist, 
that  little  attention  is  being  paid  to  the  conditions  under  which  methods  are  appropriate,  and  that  often  it  is  unclear  how 
the  products  of  cognitive  task  analysis  should  be  used. 

1.  INTRODUCTION 

As  part  of  its  Programme  of  Work,  RSG.27  on  Cognitive  Task  Analysis  has  undertaken  the  task  of  reviewing  existing 
cognitive  task  analysis  techniques  and  computer  tools. 

RSG.27  has  restricted  itself  to  a  "review  of  reviews",  in  order  to  keep  the  work  manageable.  Therefore,  descriptions  of 
individual  methods  and  techniques  were  explicitly  excluded  from  the  review. 

A  literature  search  was  conducted,  starting  with  references  from  1990  onwards.  Twenty  reviews  were  deemed  useful  for 
further  reading.  A  common  framework  was  adopted  for  summarizing  the  reviews.  The  categories  comprising  the 
framework  were  the  following: 

(i)  full  citation 

(ii)  indication  of  the  year  the  review  was  covering  the  literature  to 

(iii)  short  description  of  the  contents  of  the  review 

(iv)  definition  of  CTA  provided  by  the  review 

(v)  depth  of  description  of  methods  or  techniques  discussed 

(vi)  applications  of  interest  to  the  author(s) 


1  This  chapter  is  an  expanded  version  of  Chapter  28  in  J.M.C.  Schraagen,  S.E.  Chipman,  and  V.L.  Shalin  (Eds.), 
Cognitive  task  analysis  (in  press).  Mahwah,  NJ:  Lawrence  Erlbaum  Associates.  Published  by  permission  of  Lawrence 
Erlbaum  Associates,  Inc. 
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(vii)  apparent  theoretical  perspective  of  the  author(s) 

2.  RESULTS 

Table  I  presents  the  outcome  of  the  literature  review,  with  the  exception  of  the  description  of  the  contents  of  the  review. 
A  full  description  of  all  reviews  can  be  found  after  the  recommendations. 


Table  I:  references  in  chronological  and  alphabetical  order 


Reference 

Coverage 

of 

literature 
up  to ... 

Definition  of 

CTA 

Depth  of 
description 

Applications  of 
interest 

Theoretical 

perspective 

Grant  &  Mayes 
(1991) 

1990 

Analysis  dealing 
with  information 
flow  and  usage 

Shallow 

Human-computer 
interaction  in 
complex  systems 

cognitive 

psychology 

/cognitive 

engineering 

Reitman  Olson 
&  Biolsi  (1991) 

1989 

No 

Shallow  for  direct 
methods;  detailed 
for  indirect 
methods 

General 

experimental 

cognitive 

psychology 

Wilson  &  Cole 
(1991) 

1989 

No 

Shallow 

Instructional  design 

instructional 

design 

Aim  (1992) 

1991 

No 

Shallow 

General 

ecological 
psychology; 
systems  thinking 

Kirwan  & 
Ainsworth 
(1992) 

1992 

Range  of 
approaches  used 
for  looking  at 
mental  internal 
events  or 
knowledge 
structure 

Detailed,  in-depth, 
practical 

General,  broad 
range  of 
applications 

applied 

ergonomics 

Redding  (1992) 

1991 

Determines  mental 
processes  and 
skills  required  to 
perform  a  task  at 
high  proficiency 
levels 

Moderate  detail, 
but  insufficient  for 
novice  analysts 

job  training 

cognitive 

psychology 

Benysh, 

Koubek,  & 
Calvez  (1993) 

1991 

No 

Shallow 

Human-Computer 
Interaction  design 

applied 

psychology 

Means  (1993) 

1 

1988 

Tasks  in  which  the 
critical  steps  are 
cognitive 
operations  rather 
than  observable 
behaviors 

Detailed  for  two 
cases  (avionics 
troubleshooting 
and  air  traffic 
control) 

Training 

mainstream 
American 
cognitive  science 

Williams  & 
Kotnur  (1993) 

1991 

No 

Detailed, 
particularly  for 
machine-aided 
methods 

Intelligent  tutoring 
systems,  but  not 
limited  to  that  area 

Carnegie  Mellon 
University-style 
cognitive 
modeling 

Cooke (1994) 

1994 

No 

Shallow 

Expert  systems 

experimental 

cognitive 

psychology 

Crandall,  Klein,  1993  Methods  for  Shallow,  but  Training  naturalistic 

Militello  &  identifying  accompanying  development  decision  making 

Wolf  (1994)  cognitive  task  CD-ROM  is  more 


requirements  for  detailed 
tasks  that  involve 
true  expertise 
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Reference 

Coverage 

of 

literature 
up  to ... 

Definition  of 

CTA 

Depth  of 
description 

Applications  of 
interest 

Theoretical 

perspective 

Essens, 

Fallesen, 
McCann, 
Cannon- 
Bowers  & 

Dorfel  (1994) 

1993 

CTA  seeks  to 
describe,  in 
cognitive  terms, 
how  goals  and 
tasks  are 
accomplished 

concise;  more 
informative  about 
elicitation 
techniques  than 
about  entire 
approaches 

Design  of  decision 
aids,  but  not  limited 
to  this  area 

transitional 

between 

behavioral  and 

modem 

cognitive 

perspectives 

John  &  Kieras 
(1994) 

1994 

Analysis  of 
knowledge  of  how 
to  do  a  task  in 
terms  of  the 
components  of 
goals,  operators, 
methods,  and 
selection  rules 

Detailed,  but 
insufficient  for 
novice  analysts 
who  want  to  leam 
how  to  use  GOMS 

Human-Computer 

Interaction 

Carnegie  Mellon 
University-style 
cognitive 
modeling 

Merkelbach  & 

Schraagen 

(1994) 

1993 

Integrated  use  of 
task  modeling, 
knowledge 
modeling  and 
cognitive 
modeling 

Detailed  for  HTA, 
KADS  and 

GOMS;  shallow 
for  other 
techniques 

General; 
connections 
between  application 
areas  and  modeling 
perspectives  are 
proposed 

Carnegie  Mellon 

University-style 

cognitive 

modeling; 

generic  task 

method; 

practical  task 

analytic 

behavioral 

approaches 

Whitefield  & 
Hill  (1994) 

1993 

No 

Shallow 

Human-Computer 
Interaction  design 

computer 
systems 
engineering; 
software  design 

Dehoney 

(1995) 

1992 

Analysis  of 
expert's  mental 
model  of  the 
problem 

Detailed  on 

elicitation 

techniques 

Instructional  design 

cognitive 

psychology 

DuBois  & 

Shalin  (1995) 

1993 

No 

Detailed  on 
knowledge 
representation  used 
and  on 

combination  of 
different  methods 

job  knowledge 
testing;  training 
needs  analysis  and 
design 

personnel 
psychology  and 
cognitive  science 

Hall,  Gott,  & 
Pokomy  (1995) 

1990 

Analysis  of 

underlying 

psychological 

processes  and 

knowledge 

structures 

Detailed  for  PARI 
methodology; 
moderate  detail  for 
four  alternative 
approaches 

Training  of 
cognitively  complex 
tasks  (in  particular, 
troubleshooting) 

cognitive  science 
and  cognitive 
psychology 

Hoffman, 

Shadbolt, 

Burton  &  Klein 
(1995) 

1992 

No 

Shallow 

Mainly  expert 
systems 

social, 

organizational, 
and  experimental 
psychology 

Gordon  &  Gill 
(1997) 

1994 

Theory  or  model 
of  how  experts 
perform  their  tasks 

Shallow 

General 

cognitive 

psychology 

Table  I  shows  that  the  20  reviews  cover  the  literature  up  to  1994.  Eleven  reviews  provide  a  definition  of  CTA,  whereas 
9  do  not  provide  a  definition  at  all.  Nine  reviews  are  shallow  in  their  description  of  particular  techniques:  in  these  cases 
the  reader  should  consult  the  primary  sources  that  are  being  reviewed.  However,  eleven  reviews  provide  detailed 
information  on  a  limited  number  of  techniques.  The  applications  of  interest  to  the  authors  appear  to  be  mostly  training 
and  instructional  design  (8  cases),  followed  by  human-computer  interaction  (4  cases)  and  expert  systems  (3  cases).  The 
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apparent  theoretical  perspective  of  the  authors  is  predominantly  a  cognitive  psychological  one  (7  cases),  followed  by  a 
cognitive  science  perspective  (e.g.  Carnegie  Mellon  University-style  production  system  modeling). 

An  analysis  of  the  reviews  themselves  shows  that  a  very  large  number  of  particular,  rather  limited  methods  are 
described  over  and  over  again.  But  little  is  said  about  how  these  can  be  effectively  orchestrated  into  an  approach  that 
will  yield  a  complete  analysis  of  a  task  or  job.  Little  is  said  about  the  conditions  under  which  an  approach  or  method  is 
appropriate.  Clearly,  the  relevant  conditions  which  need  to  be  considered  include  at  least  the  type  of  task  being 
analyzed,  the  purpose  for  which  the  analysis  is  being  done  (Human-Computer  Interaction  design,  training,  testing, 
expert  system  development),  and  the  resources  available  for  the  analysis,  particularly  the  type  of  personnel  available  to 
do  the  analysis  (cognitive  scientists,  cognitive  psychologists,  educational  specialists,  subject-matter  experts).  The 
literature  is  also  very  weak  when  it  comes  to  specifying  the  way  in  which  the  products  of  task  analysis  should  be  used  in 
designing  either  training  or  systems  with  which  humans  will  interact. 

Based  on  the  literature  review,  the  following  issues  were  identified  by  RSG.27  as  timely  and  important  to  investigate: 

1 .  Validity  and  reliability  of  CTA  techniques  and  results 

2.  CTA  for  new  tasks/systems 

3.  CTA  for  tasks  that  are  difficult  to  verbalize  (e.g.,  spatial  tasks) 

4.  Requirements  for  analysts  (required  level  of  training  and  education) 

5 .  Conditions  under  which  particular  techniques  should  be  employed 

6.  Relation  between  the  purpose  of  CTA  and  the  results  of  the  activity  (do  different  types  of  applications  require 
different  results?  E.g.,  training  de  novo,  retraining,  proficiency  testing,  Human-Computer  Interaction,  system 
(re)design) 

7.  CTA  for  team  tasks 

8.  Relationship  of  CTA  techniques  to  theories  of  cognition 

9.  Individual  differences  (culture,  nationality)  and  their  implication  for  solutions 

10.  Effects  of  environmental  stressors  on  the  conduct  of  CTA  (does  CTA  need  to  incorporate  the  stressors  in  the 
environment  in  which  the  operator  works?) 

1 1 .  Selecting/defining  expertise 

12.  Human  Operator/Performance  Modeling  Systems 

13.  CTA  for  safety-critical  systems;  Human  Reliability  Analysis 

RECOMMENDATIONS 

1.  RSG.27  recommends  the  organization  of  a  workshop  with  experts  in  the  field  of  Cognitive  Task  Analysis.  The 
goal  of  the  workshop  would  be  to  address  the  issues  identified  by  RSG.27  and  thus  advance  the  field  of  Cognitive  Task 
Analysis. 

2.  RSG.27  recommends  that  the  workshop  be  centered  around  the  following  four  themes: 

1 .  What  is  the  purpose  of  a  Cognitive  Task  Analysis  and  what  results  are  generated  to  achieve  this  purpose? 

2.  What  integrated  approaches  are  used  by  analysts  and  how  should  these  be  evaluated? 

3.  What  are  the  scientific  underpinnings  of  Cognitive  Task  Analysis? 

4.  What  methods/topics/approaches  are  recommended  for  further  research? 

3.  RSG.27  recommends  that  the  results  of  the  workshop  be  made  available  widely,  preferably  in  an  edited  book 
format. 
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S.  Grant  &  T.  Mayes  (1991).  Cognitive  task  analysis?  In  G.R.S.  Weir  &  J.L.  Alty  (Eds.),  Human-computer  interaction 
and  complex  systems  (pp.  147-167).  London:  Academic  Press 

Literature  covered  up  until  1990,  most  references  are  from  the  eighties. 

The  focus  in  this  chapter  is  on  cognitive  task  analysis  for  human-computer  interaction  in  complex  systems.  The  authors 
argue  that  since  humans  will  continue  to  play  a  key  role  in  controlling  or  supervising  complex  systems,  these  systems 
need  to  be  designed  taking  into  account  the  human's  information  needs.  "A  CTA  will  properly  involve  an  analysis  of  the 
information  flow  and  the  required  knowledge  states  throughout  the  entire  system"  (p.  149).  The  authors  distinguish 
three  approaches  to  CTA: 

1 .  An  analysis  of  a  task  in  terms  which  relate  to  human  cognition  (decomposition  formalisms) 

2.  Theories  and  models  of  human  cognition  which  specify  the  terms  in  which  the  task  has  to  be  analyzed 

3 .  Observations  on  salient  features  of  human  cognition  in  complex  processes. 

Examples  of  the  first  approach  that  are  discussed  are:  GOMS,  Cognitive  Complexity  Theory,  Command  Language 
Grammar,  and  Task-Action  Grammars.  The  main  shortcoming  of  these  formalisms,  according  to  the  authors,  is  that 
they  have  been  restricted  to  simpler  systems  and  highly  proceduralized  behavior.  The  authors  are  also  skeptical  about 
representing  knowledge  in  terms  of  an  unrestricted  production  system,  precisely  because  multiple  solutions  are  equally 
possible.  It  is  therefore  unclear  whether  any  particular  production  system  bears  any  resemblance  to  actual  cognitive 
behavior. 

The  models  of  cognition  that  are  discussed  as  examples  of  the  second  approach  are:  The  Model  Human  Processor, 
Programmable  User  Models,  ACT*,  Interacting  Cognitive  Subsystems.  Again,  these  models  are  more  suitable  for 
analyzing  short-term  simple  tasks  than  longer-tenn  subtler  cognitive  abilities  involving  problem  solving  or  decision 
making.  Models  such  as  ACT*  (and  SOAR)  need  a  way  of  finding  those  production  rules  which  best  represent  a 
particular  human  approach  to  a  particular  task.  These  models  do  not  help  to  focus  an  approach  to  analysis,  but  rather 
leave  this  aspect  of  the  analysis  open  (in  general,  it  is  possible  to  analyze  a  task  in  terms  of  production  rules  in  many 
widely  differing  ways).  The  authors  are  clearly  in  favor  of  Interacting  Cognitive  Subsystems. 

Among  the  important  features  of  cognition  in  complex  systems  mentioned  are  the  Rasmussen  "Skills,  Rules,  and 
Knowledge"  framework  ,  Roth  and  Woods'  "Mapping  Cognitive  Demands"  framework  and  Moray's  "Quasi¬ 
independent  subsystems".  Although  the  authors  find  the  Rasmussen  framework  useful,  it  does  not  amount  to  a  complete 
cognitive  task  analysis  technique.  Roth  and  Woods’  attempt  to  map  cognitive  demands  independently  of  representation 
and  cognitive  agent  is  difficult  given  the  possibility  of  individual  differences  and  uncertainty  about  how  the  domain 
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problem  is  to  be  described  in  the  first  place.  The  problem  with  Moray's  analysis  is  that  he  offers  no  evidence  about  the 
extent  to  which  actual  models  of  operators  match  up  with  the  methodical  analyses. 

The  authors  conclude  from  their  review  that  none  of  the  approaches  discussed  are  fully  adequate  for  a  CTA  in  the 
context  of  complex  systems:  they  either  lack  sufficient  input  from  cognitive  psychology,  or  they  lack  sufficient 
relevance  to  the  realities  of  complex  tasks,  or  both.  The  aim  of  CTA  must  be  to  integrate  an  investigation  of  tasks 
independently  of  human  cognition  and  an  investigation  of  cognitive  processes  involved  in  complex  control  tasks. 

Overall,  this  is  more  a  philosophical  than  a  practical  chapter,  that  is  also  somewhat  outdated.  The  only  practical  advice 
offered  is  "investigate  the  information  requirements  of  specific  operators  in  specific  situations"  (pp.  165-166).  The  good 
point  is  that  the  authors  are  concerned  with  actual  cognitive  behavior  as  displayed  by  human  operators.  They  offer  some 
critical  comments  about  cognitive  modeling  enterprises  that  seemingly  ignore  actual  cognitive  behavior.  In  a  current 
version  of  their  chapter,  the  authors  would  need  to  include  EPIC  (Meyer  &  Kieras)  and  ACT-R  (Anderson),  although 
many  of  their  comments  would  remain  the  same. 

Definition  of  cognitive  task  analysis:  "an  analysis  dealing  with  information  flow  and  usage"  (p.  165) 

Depth  of  description:  The  descriptions  are  quite  brief  and  readers  are  referred  to  other  sources  for  details.  The 
overarching  comments  in  this  paper  tend  to  be  more  valuable  than  the  descriptions  of  the  formalisms  or  models.  The 
descriptions  are  not  useful  for  practitioners  who  want  to  apply  techniques. 

Applications  of  interest  to  the  authors:  human-computer  interaction  in  complex  systems 

Apparent  perspective  of  the  authors:  cognitive  psychology/cognitive  engineering 

J.  Reitman  Olson  &  K.J.  Biolsi  (1991)  Techniques  for  representing  expert  knowledge.  In:  K.  Anders  Ericsson  &  J. 
Smith  (Eds),  Toward  a  general  theory  of  expertise  (pp.  240-285).  Cambridge,  England:  Cambridge  University  Press. 

This  paper  cites  a  great  deal  of  methodological  work  that  extends  quite  far  back;  it  seems  to  provide  coverage  up  to 
about  1988-89. 

This  paper  reviews  quite  a  number  of  knowledge  elicitation  techniques  along  with  their  associated  analysis  methods  and 
resulting  output  representations  of  knowledge.  The  techniques  are  divided  into  direct  (interview,  think-aloud, 
observation,  interruption  analysis,  commentary,  drawing  of  groupings  in  a  spatial  array,  card  sorting)  and  indirect 
methods  (direct  judgments  of  similarity  or  relatedness,  confusion  probabilities,  co-  occurrence  probabilities,  repertory 
grid  and  recall  orders).  Useful  diagrams  bring  out  the  relationships  among  the  members  of  these  two  families  of 
methods.  The  direct  methods  are  not  treated  in  any  detail:  they  are  described  as  a  way  to  discover  the  vocabulary  and 
concepts  in  an  area  of  expertise  and  as  the  only  way  to  uncover  problem-solving  strategies.  There  is  no  discussion,  for 
example,  of  ways  of  designing  or  structuring  interviews.  Some  comments  are  made  about  the  domains  in  which  think 
aloud  methods  are  suitable  (those  naturally  involving  verbal  concepts)  and  not  suitable  (perceptual,  motor,  highly 
automatized.)  In  contrast,  the  indirect  methods  are  discussed  in  considerable  detail  both  with  respect  to  procedures  and 
assumptions  about  the  underlying  knowledge  representations  and  assumptions  about  the  way  in  which  subjects  may  be 
generating  the  data  collected  by  the  method  from  the  underlying  representation.  A  table  summarizing  experience  with 
these  methods  (p.  274)  reveals  that  most  are  suited  when  it  is  reasonable  to  assume  a  hierarchical  representation  of 
concepts.  The  text  also  comments  that  multidimensional  scaling  has  proved  more  suitable  to  the  analysis  of  perceptual 
similarities,  whereas  the  other  methods  seem  more  appropriate  for  semantic  concepts.  Some  concern  is  expressed  about 
the  psychological  reality  of  the  representations  emerging  from  these  methods  and  about  the  desirability  of  obtaining 
convergent  evidence  from  multiple  methods.  A  useful  general  comment  is  made  about  the  importance  of  paying 
attention  to  the  extent  to  which  the  data  collection  task  appears  to  be  easy  and  natural  for  the  subjects.  There  are  also  a 
few  useful  comments  about  the  orchestration  of  these  different  approaches  into  a  total  approach  —  the  use  of  interviews 
to  identify  important  concepts,  the  use  of  indirect  methods  to  explore  relations  among  these  concepts,  and  the  possible 
integration  of  think  aloud  or  similar  methods  into  indirect  data  collection  techniques  in  order  to  ease  interpretation  of 
the  results.  However,  some  points  that  are  rather  obvious  from  the  discussion  are  not  brought  out  strongly  and 
explicitly.  Although  the  paper  begins  by  pointing  out  the  richness  of  true  expert  knowledge,  the  implications  of  this  are 
not  brought  out.  In  particular,  the  problem  of  the  small  number  of  concepts  that  can  typically  be  studied  with  the 
indirect  techniques  is  not  discussed.  Also,  despite  the  emphasis  on  the  importance  of  the  nature  of  the  representations 
and  of  the  assumptions  on  this  point,  nothing  is  said  about  a  direct  focus  on  this  issue. 

Definition  of  cognitive  task  analysis:  the  term  cognitive  task  analysis  is  not  prominent  in  this  paper  and  no  definition  of 
it  is  given. 

Depth  of  description:  As  noted  above,  the  description  of  so-called  direct  methods  is  shallow,  while  the  description  of 
indirect  experimental  methods  is  quite  detailed  and  supplemented  by  useful  comparisons  among  them  and 
methodological  critiques. 
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Tasks  of  interest:  This  review  is  quite  general,  with  no  focus  on  a  particular  type  of  expertise  or  application. 

Apparent  perspective  of  the  authors:  Although  Reitman  Olson  has  been  involved  in  applied  work,  especially  in  Human- 
Computer  Interaction,  I  would  characterize  this  review  as  being  written  from  the  perspective  of  an  experimental 
cognitive  psychologist  with  a  basic  research  orientation. 

Wilson,  B.,  &  Cole,  P.  (1991).  A  review  of  cognitive  teaching  models.  Educational  Technology,  Research  and 
Development,  39(4),  47-64. 

Dates  covered:  This  review  [of  cognitive  teaching  models,  not  cognitive  task  analysis]  spans  the  period  from  1980  to 
1989. 

Abstract:  This  article  reviews  (from  primarily  an  instructional-design  perspective)  nine  teaching  programs  developed  by 
cognitive  psychologists  over  the  last  dozen  years  or  so.  Among  these  models,  Collins  cognitive  apprenticeship  model 
has  the  most  explicit  prescriptions  for  instructional  design.  This  cognitive  apprenticeship  model  is  analyzed,  then 
components  of  the  model  are  used  as  an  organizing  framework  for  understanding  the  remaining  models.  Differences  in 
approach  are  noted  between  traditional  ID  prescriptions  and  the  cognitive  teachings  models.  Surprisingly,  no  design 
strategies  were  found  to  be  common  to  all  the  model  programs.  Key  differences  among  programs  included:  (a)  problem 
solving  versus  skill  orientations,  (b)  detailed  versus  broad  cognitive  task  analysis,  (c)  learner  versus  system  control,  and 
(d)  error-restricted  versus  error-driven  instruction.  The  article  concludes  with  an  argument  for  the  utility  of  continuing 
dialog  between  cognitive  psychologists  and  instructional  designers. 

Definition  of  CTA:  Only  one  small  section  in  the  paper  actually  addresses  issues  related  to  cognitive  task  analysis  (pp. 
59-60).  Very  generally,  the  main  products  of  eta  include:  (a)  the  representation  of  the  content/knowledge  or  skills  taught 
by  whatever  methods,  and  (b)  the  instructional  materials  and  interactions. 

Depth  of  description:  Again,  this  review  was  mostly  on  discussing  a  variety  of  "current"  cognitive  teaching  models 
(e.g.,  anchored  instruction,  qualitative  mental  models).  Each  one  of  them  had  a  different  requirement  with  regard  to  the 
grain  size  of  the  instructional  content,  from  low-level  (e.g.,  Anderson's  ITS,  representing  content  as  productions)  to 
high-level  (e.g.,  mental  models). 

Applications  of  interest  to  the  authors:  Their  orientation  is  clearly  instructional  design  thus  their  interest  appears  to  be 
finding  pedagogical  prescriptions.  But  that  doesn't  happen.  Rather,  the  strength  of  this  paper  is  in  terms  of  its  clean 
summaries  of  some  of  the  most  influential  approaches  to  teaching  that  arose  during  the  80s. 

Apparent  theoretical  perspective  of  the  author:  Instructional  Design. 

I.  Aim  (1992).  Cognitive  tasks:  A  meta-analysis  of  methods(FOA  report  C  50098.5.5).  Sundbyberg,  Sweden:  National 
Defense  Research  Establishment,  Department  of  Human  Studies. 

This  review  covers  the  literature  up  until  1991. 

This  report  does  not  frilly  deliver  what  it  promises-namely  a  meta-analysis  of  methods  (for  cognitive  task  analysis). 
Only  two  methods  are  discussed  in  some  detail:  the  Generic-Error  Modeling  System  (GEMS)  and  Hierarchical  Task 
Analysis.  The  major  claim  of  the  report  is  that  these  and  other  methods  for  cognitive  task  analysis  generally  ignore  the 
context  in  which  cognitive  behavior  takes  place.  This  is  similar  to  Simon's  claim  that  the  task  environment  to  a  large 
extent  determines  the  representations  used  by  problem  solvers.  For  instance,  according  to  Aim  "it  is  not  the  same  thing 
to  drive  alone  on  a  road  a  sunny  day  and  drive  a  car  during  a  rash  hour  a  rainy  day"  (p.  11).  The  task,  driving  a  car,  is 
the  same  on  both  occasions,  but  the  change  in  context  leads  to  quite  different  demands  on  the  driver  and  consequently  to 
quite  different  driving  behavior. 

The  report  distinguishes  between  two  types  of  contexts:  physical  and  social  structures.  In  the  first,  a  cognitive  system 
interacts  with  a  non-cognitive  system;  in  the  second,  two  or  more  cognitive  systems  interact.  Furthermore,  two  different 
types  of  tasks  are  distinguished:  iterative  or  repetitive  tasks,  and  open-ended  or  dynamic  tasks.  Iterative  tasks  in  either  a 
physical  or  social  structure  require  a  different  approach  when  being  analyzed  than  open-ended  tasks.  Iterative  tasks 
require  that  the  analyst  find  out  what  instruction  is  valid  for  task  performance,  what  the  performer's  mental 
representation  of  the  task  and/or  the  instruction  is,  and  what  biases  cause  the  two  to  differ.  Open-ended  tasks  performed 
in  a  physical  or  social  structure  require  the  observer  to  find  out  the  system's  function,  the  organization  of  the  task,  and 
the  system's  relation  to  the  environment.  Next,  a  context  for  a  particular  task  performance  is  selected  and  studied  during 
some  period  of  time  ('space-time  structure').  This  analysis  should  be  checked  with  the  performers.  Finally,  the  mental 
representations  of  the  task  and  the  space-time  structure  should  be  examined. 
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The  major  conclusion  of  the  report  is  that  there  does  not  exist  any  universal  method  which  can  analyse,  describe,  and 
explain  all  cognitive  tasks  performed  in  different  contexts.  Therefore,  prior  to  analyzing  a  particular  task's  demands  on 
the  cognitive  system,  it  is  extremely  important  to  differentiate  between  various  kinds  of  purposes,  situations,  tasks, 
structures  and  organizations. 

No  explicit  definition  of  cognitive  task  analysis  is  provided  in  the  report.  However,  cognition  is  viewed  as  "an  open 
self-regulating  system  characterized  by  feedback  loops"  (p.  27). 

Depth  of  description:  only  GEMS  and  HTA  are  discussed  in  some  detail,  but  still  fairly  superficially. 

Applications  of  interest:  the  report  does  not  focus  on  any  particular  application  areas.  Its  conclusions  are  deemed 
relevant  for  all  application  areas. 

Apparent  theoretical  perspective  of  author:  ecological  psychology  (Bateson),  systems  thinking  (Laszlo).  Although  the 
author  does  not  refer  to  this  theoretical  area,  she  seems  to  be  close  to  situated  cognition. 

Kirwan,  B.  &  L.K.  Ainsworth  (Eds.)  (1992),  A  guide  to  task  analysis.  London:  Taylor  &  Francis.  417  pages. 

This  book  covers  the  literature  up  until  1992.  A  British  Task  Analysis  Working  Group,  consisting  of  members  from 
industries,  universities  and  consultancies  produced  the  guide  from  1989  to  1991. 

Although  it  is  sometimes  claimed  (e.g.,  by  Gordon  &  Gill,  1997)  that  this  book  solely  contains  traditional  behavioral 
task  analysis  methods,  this  is  certainly  not  the  case.  Various  methods  that  are  usually  claimed  to  be  'cognitive'  in  nature, 
are  discussed,  for  instance:  critical  incident  technique,  GOMS,  Hierarchical  Task  Analysis,  structured  interviews,  verbal 
protocols,  walk-through/talk-through  analyses,  withheld  information.  Admittedly,  the  authors  very  briefly  list  a 
technique  they  themselves  describe  as  'cognitive  task  analysis'  on  page  392  only  (this  is  the  TAKD  technique).  Since  the 
only  reference  to  cognitive  task  analysis  in  the  book  is  to  this  short  paragraph,  the  authors  must  view  the  other  methods 
listed  above  as  non-cognitive.  This  is  strange  because  without  doubt  both  GOMS  and  verbal  protocols  are  purely 
cognitive  in  nature.  The  Guide  is  stronger,  then,  on  the  behavioral  techniques,  although  the  description  of  verbal 
protocols,  for  instance,  is  very  useful  and  balanced. 

The  Guide  contains  descriptions  of  25  techniques  and  illustrates  the  practical  use  of  these  techniques  in  10  case  studies. 
The  case  studies  clearly  show  'lessons  learned'  and  are  very  powerful  illustrations  of  what  task  analysis  techniques  can 
provide  for  an  organization. 

Definition  of  cognitive  task  analysis:  "Cognitive  task  analysis  covers  a  range  of  approaches  used  for  looking  at  mental 
(hence  cognitive)  internal  events  or  knowledge  structure"  (p.  392). 

Depth  of  description:  Excellent,  in-depth  descriptions  of  25  techniques,  with  practical  advice,  resources  required,  links 
with  other  techniques,  advantages  and  disadvantages.  Key  references  to  primary  sources  are  provided. 

Applications  of  interest  to  authors:  allocation  of  function,  person  specification,  staffing  and  job  organization,  task  and 
interface  design,  skills  and  knowledge  acquisition,  performance  assurance.  The  case  studies  are  largely  focused  the 
above  issues  in  industrial  settings,  e.g.,  nuclear  power  plants  and  process  control. 

Apparent  theoretical  perspective  of  the  authors:  Applied  ergonomics. 

Redding,  R.  E.  (1992).  A  standard  procedure  for  conducting  cognitive  task  analysis.  McLean,  VA:  Human  Technology, 
Inc.  (ERIC  Document  Reproduction  Service  No.  ED  340  847). 

Dates  covered:  This  paper  does  not  provide  a  review  of  published  cognitive  task  analysis  research,  but  provides 
guidelines  for  conducting  a  cognitive  task  analysis.  A  "Suggested  References"  section  includes  several  citations  for 
more  detailed  information  about  approaches  to  CTA  within  an  ISD  framework  (dated  from  1986  to  1991). 

Abstract:  The  Instructional  Systems  Development  (ISD)  model  has  been  widely  used  and  found  effective  for  analysis 
and  training  of  psychomotor  skills.  It  includes  five  phases:  (Task)  Analyze,  Design,  Develop,  Implement,  and  Control. 
However,  the  ISD  model  has  not  been  effective  for  jobs  requiring  complex  cognitive  skills  (e.g.,  decision-making, 
problem-solving,  attention-switching,  and/or  the  effective  organization  and  retrieval  of  large  amounts  of  knowledge). 
As  Redding  points  out,  cognitive  methods  need  to  be  incorporated  into  ISD  task  analysis,  and  this  paper  provides  a 
standard  procedure  for  conducting  such  an  analysis. 
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The  guidelines  given  in  this  document  provide  a  method  for  accomplishing  a  cognitive  task  analysis  and  preparing  the 
deliverables  (i.e.,  research  proposal,  optional  interim  report,  and  final  report).  Redding  outlines  critical  actions  which 
are  described,  below.  Additionally,  he  includes  some  sections  regarding  what  should  be  reported  in  documentation  (e.g., 
proposals  and  reports),  data  collection  methods  used  for  cognitive  task  analysis  (i.e.,  protocol  analysis,  psychological 
scaling,  performance  modeling,  and  observation  of  job  performance  and  interviewing),  controlled  experimental  design, 
measures  and  experimental  treatments,  data  analysis  methods  and  validation  of  results.  The  5  critical  actions  are:  1. 
Identify  key  job  tasks  and  training  issues.  This  includes  determining  the  goals  and  foci  of  the  task  analysis,  specific 
training  problems,  justifications  for  job  task  selections  (e.g.,  tasks  where  frequent  errors  are  made,  high  performance 
skills,  tasks  requiring  large  amounts  of  knowledge),  what  is  already  known  about  the  nature  of  expertise  in  the  job  task 
(e.g.,  through  manuals,  operating  procedures,  expert  interviews),  and  describing  the  population  to  be  used  in  the 
analysis  (i.e.,  defining  "expert"  on  the  task,  determining  sample  size,  determining  the  range  of  ability  or  experience 
levels  in  sample).  2.  Develop  visual  representations  of  knowledge  structures.  One  should  present  the  organization  of 
experts'  (and/or  less  experienced  persons')  knowledge  structures  about  the  task  in  a  visual  model  using  a  network  of 
linked  concepts,  a  tree  diagram,  or  a  schema  depicting  interrelationships  between  knowledge.  3.  Describe  cognitive 
processes  underlying  performance.  This  includes  the  skills,  conceptual  and  procedural  knowledge,  and  learning  and 
performance  strategies  that  result  in  effective  task  performance.  For  example,  effective  mental  models  for  the  task, 
types  of  skills  required  for  competent  performance,  effective  cognitive-optimizing  strategies,  visual  representations, 
conceptual  and  procedural  knowledge  needed  for  the  task  as  a  whole  and  as  they  interrelate,  important  task  components 
or  subgoals,  and  strategies,  heuristics,  algorithms,  or  aids  used  for  learning  or  during  job  performance  (e.g.,  production 
rales,  IF. ..THEN  rales,  GOMS,  decision  trees).  4.  Identify  differences  between  experts  &  novices.  This  includes 
differences  in  knowledge  structures  (optional),  skills,  conceptual  and  procedural  knowledge,  and  learning  and 
performance  strategies.  This  also  aids  in  determining  effective  job  performance  skills,  knowledge  and  knowledge 
structures.  5.  Determine  implications  of  results  for  design  phase.  The  results  of  a  cognitive  task  analysis  (i.e.,  curricular 
guts)  should  allow  one  to  recommend  a  training  structure  (e.g.,  improve  understanding  of  interrelationships,  improve 
organization  to  support  problem-solving,  diagnose  and  remedy  novice  errors),  a  training  sequence  (i.e.,  according  to 
skill,  knowledge,  and  knowledge  structure  progressions),  and  the  instructional  setting,  media,  and  strategies  which 
facilitated  learning. 

Definition  of  CTA:  "Cognitive  task  analysis  determines  the  mental  processes  and  skills  required  to  perform  a  task  at 
high  proficiency  levels  and  the  changes  that  occur  as  the  skills  develop.  Cognitive  task  analysis  helps  determine  how  a 
task  is  learned,  how  learning  can  be  facilitated,  and  what  cognitive  processes  underlie  and  support  effective  job 
performance.  The  information  is  then  used  in  the  design  phase  to  determine  the  most  effective  manner  of  training 
employees  to  achieve  high  levels  of  proficiency  in  job  performance."  (p.  3) 

Depth  of  description:  Due  to  its  guideline/outline  format,  this  paper  helps  one  consider  the  task  from  start  to  finish  by 
mentioning  every  aspect  of  the  analysis.  The  critical  actions  (given  above)  are  described  quite  well.  Questions  to  ask 
(e.g.,  for  conducting  the  initial  research  into  the  domain)  and  methods  to  use  (e.g.,  to  learn  about  the  expertise  in  the  job 
task)  are  listed,  but  the  suggested  reading  would  be  required  for  more  specific  details  of  how  to  go  about  these  methods, 
if  one  is  inexperienced  in  CTA.  These  readings  are  given  on  pages  5  and  6. 

Applications  of  interest  to  the  authors.  These  guidelines  are  quite  general,  but  specify  their  use  for  job  training  and 
research  proposals  and  deliverables.  This  paper  is  a  great  starting  point  for  practitioners  interested  in  organizing  and 
conducting  cognitive  task  analysis. 

Apparent  theoretical  perspective  of  the  author:  cognitive  psychology  perspective,  elaborating  on  the  existing  "Analyze" 
of  the  ISD  model.  His  guidelines  also  include  basic  research  methods  to  use  in  conducting  a  cognitive  task  analysis. 

D.V.  Benysh,  R.J.  Koubek  &  V.  Calvez.  (1993)  A  comparative  review  of  knowledge  structure  measurement  techniques 
for  interface  design.  International  Journal  of  Human-Computer  Interaction,  5(3),  211-237. 

Cites  literature  through  1990-1991. 

The  authors  state  that  the  purpose  of  their  paper  is  to  provide  researchers  in  human  computer  interaction  with  a 
comparative  review  of  available  techniques  and  to  summarize  their  potential  application  for  aiding  designers  at  various 
stages  of  interface  design.  They  classify  techniques  into  three  classes:  verbal  reports,  clustering  techniques  and  scaling 
methods.  (In  fact,  the  latter  two  classes  are  not  that  cleanly  separated  in  the  actual  review.)  For  each  class,  they  discuss 
the  manner  of  data  gathering  and  concept  elicitation,  the  method  used  to  derive  the  knowledge  structure  and  the  analysis 
procedures  used.  The  category  of  verbal  reports  includes  interviewing,  questionnaires,  interruption  analysis  and 
protocol  analysis.  These  are  discussed  very  briefly  but  the  authors  do  make  an  important  point  about  the  significant  role 
of  the  analyst  in  determining  how  the  knowledge  structures  will  be  represented.  They  point  out  that  cognitive  modeling 
schemes  such  as  GOMS,  Task  Action  Grammar,  or  production  rales  are  sometimes  used  in  the  representation.  In 
discussing  clustering  methods,  the  authors  give  much  more  attention  to  the  problem  of  developing  the  list  of  concepts  to 
be  studied  than  do  most  reviewers,  and  they  review  several  methods  for  this  purpose.  The  methods  of  cluster  elicitation 
discussed  are  card  sorting,  ordered  recall,  closed  curve  analysis  (drawing  curves  around  things  that  "go  together"),  and 
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spatial  reconstruction.  They  point  out  that  closed  curve  analysis  has  revealed  overlapping  clusters,  whereas  most 
methods  of  analysis  assume  discrete  or  hierarchical  clusters.  Scaling  methods  involve  the  same  problem  of  creating 
concept  lists.  The  methods  of  obtaining  concept  similarity  that  are  discussed  are  similarity  ratings,  repertory  grid, 
co-occurrence  analysis  (e.g.  in  card  sorting),  proximity  in  recall.  Methods  of  analysis  for  these  data  include  conversion 
to  general  weighted  networks  (e.g.  by  Pathfinder),  multidimensional  scaling,  and  hierarchical  clustering.  They  briefly 
discuss  the  problem  of  comparing  the  resulting  representations  across  individuals.  Throughout  the  paper  a  few  brief 
remarks  are  made  about  how  this  information  could  be  useful  in  interface  design  (e.g.  in  grouping  menu  items).  They 
comment  that,  with  the  exception  of  the  verbal  report  methods,  these  methods  primarily  get  at  declarative  knowledge. 
Explicit  definition  of  cognitive  task  analysis:  there  is  none  given;  in  fact,  the  term  they  use  is  "knowledge  structure 
measurement  techniques".  They  are  motivated  by  the  fact  that  knowledge  structures  have  been  shown  to  be  predictors 
of  operator  performance. 

Depth  of  description:  For  the  most  part,  the  descriptions  are  quite  brief  and  readers  are  referred  to  other  sources  for 
details.  The  overarching  comments  in  this  paper  tend  to  be  more  valuable  than  the  descriptions  of  data  collection  or 
analysis  techniques. 

Applications  of  interest  to  the  authors:  as  noted  above,  primarily  Human-Computer  Interaction  design.  However,  they 
also  show  interest  in  the  prediction  of  performance  from  the  measurement  of  knowledge  structures.  Koubek  has  also 
done  work  in  which  he  attempted  to  directly  train  knowledge  structures.  After  this  paper,  he  went  on  to  propose  a 
knowledge  representation  structure  that  combines  both  conceptual  and  procedural  knowledge. 

Apparent  perspective  of  the  authors:  that  of  very  applied  psychologists  who  are  involved  in  personnel  selection, 
interface  design  and  training. 

B.  Means  (1993)  Cognitive  task  analysis  as  a  basis  for  instructional  design.  In:  M.  Rabinowitz  (Ed)  Cognitive  science 
foundations  of  instruction  (p.  97-118). 

Key  references  are  concentrated  in  1988. 

This  paper  effectively  makes  the  case  for  the  need  to  focus  on  cognitive  aspects  of  many  tasks  to  supplement  the 
information  provided  by  more  traditional  behavioral  task  analyses,  using  the  example  of  air  traffic  control.  It  also  points 
out  the  problems  often  posed  by  experts'  lack  of  explicit  conscious  access  to  much  of  their  task  knowledge.  It  advocates 
a  team  approach  to  cognitive  task  analysis  in  which  the  instructional  and/or  cognitive  experts  work  with  collaborating 
subject  matter  experts  who  both  design  example  tasks  or  scenarios  to  bring  out  necessary  expertise  and  serve  as  subjects 
in  solving  problems,  providing  protocols,  etc.  Two  case  examples  are  described,  one  involving  avionics  trouble¬ 
shooting  and  the  other  air  traffic  control.  These  differ  in  greatly  in  task  pacing  and  thus  in  the  techniques  that  are 
feasible  for  studying  task  performance.  The  air  traffic  control  case  also  provides  interesting  examples  of  the  role  of 
metacognitive  knowledge  or  skills.  The  analytic  approach  draws  heavily  upon  the  tradition  of  expert-novice  studies  in 
cognitive  science  both  as  a  theoretical  foundation  and  as  a  source  of  experimental  (or  knowledge  elicitation)  techniques. 
Quite  a  number  of  such  techniques  were  used  in  the  cases  described  but  they  are  not  explicitly  called  out  or  separately 
described. 

Definition  of  cognitive  task  analysis:  No  explicit  definition  is  given  but  cognitive  task  analysis  is  described  as 
appropriate  for  tasks  in  which  the  critical  steps  are  cognitive  operations  such  as  hypothesis  formation,  judgements,  or 
problem  solving  rather  than  observable  behaviors,  (p.  101). 

Depth  of  description:  The  two  cases  are  described  in  considerable  detail,  such  that  a  cognitive  psychologist  or  cognitive 
scientist  might  well  be  able  to  generalize  the  approach  to  other  problems.  Specific  experimental  or  knowledge 
elicitation  techniques  and  the  analyses  of  the  data  they  provide  are  not  described  in  any  detail. 

Applications  of  interest  to  authors:  Cognitive  task  analysis  for  the  purpose  of  developing  military,  industrial,  and  similar 
training.  The  two  cases  concern  electronics  troubleshooting  and  civilian  air  traffic  control;  the  more  general  class  is 
tasks  that  involve  high  levels  of  expertise. 

Apparent  theoretical  perspective  of  the  author:  mainstream  American  cognitive  science,  as  evidenced  by  the  emphasis 
on  expert-  novice  studies. 

Note:  at  least  in  part,  this  is  describing  the  same  cognitive  task  analysis  approach  as  Hall,  Gott  &  Pokomy  (1995). 

K.  E.  Williams  &  T.G.  Kotnur  (1993)  Knowledge  acquisition:  A  review  of  manual,  machine-aided  and 
machine-learning  methods.  (Technical  report  on  Office  of  Naval  Research  contract  N00014-91-J-  5-1500)  Blacksburg, 
VA:  Virginia  Polytechnic  Institute  and  State  University,  Management  Systems  Laboratories.  62  pages. 
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Literature  cited  extends  to  1990-1991. 

"This  review  and  classification  of  knowledge  acquisition  methods  was  conducted  as  part  of  a  design  process  in  the 
development  of  an  automated  aid  for  creating  detailed  cognitive  models  of  procedural  tasks  consistent  with  a  production 
system  architecture  of  cognition.  The  target  application  for  the  automated  aid  was  the  authoring  of  ideal  student  models 
for  intelligent  tutoring  systems.  The  review  was  conducted  to  identify  the  various  kinds  of  methods  which  have  been 
developed  for  knowledge  acquisition  since  ideal  student  model  generation  is  a  knowledge  acquisition  process.  As  a 
result  of  this  review,  approximately  40  different  methods  were  identified  and  classified.  The  methods  are  classified 
along  a  continuum  of  the  degree  to  which  automation  is  employed  in  implementing  the  knowledge  acquisition  process. 
Three  classes  of  methods  were  defined:  manual  methods,  machine-aided  methods  and  machine  learning  methods." 
(portion  of  authors'  abstract). 

A  useful  graph  showing  the  classification  and  subclassifications  of  the  methods  reviewed,  as  well  as  naming  those 
specific  methods,  appears  on  p.  6  of  the  report.  The  review  is  more  comprehensive  than  most  because  of  its  inclusion  of 
machine-aided  and  machine  learning  methods  but  does  not  claim  to  be  exhaustive.  In  particular,  the  review  tends  to 
present  one  example  of  a  machine-aided  version  of  a  method  rather  than  discussing  all  versions  that  may  be  available. 
Some  effort  is  made  to  provide  worked  out  examples  of  the  various  fundamentally  different  methods  for  instructional 
purposes.  More  detail  is  provided  for  the  machine-aided  methods.  In  the  description  of  methods  for  getting  at 
conceptual  organization,  the  report  relies  heavily  on  Olson  &  Reuter  (1987).  It  credits  Carbonell  (1990)  and,  to  a  lesser 
extent  Michalski  (1991),  for  the  taxonomy  of  machine  learning  methods.  Among  the  cognitive  task  analysis  methods,  it 
includes  Miyake's  (1986)  "constructive  interaction"  method  for  getting  information  about  how  a  device  or  system 
works,  which  does  not  seem  to  be  mentioned  in  most  reviews.  The  paper  ends  with  a  conclusion  and  discussion  focused 
on  the  system  design  purpose  for  which  the  review  was  done,  but  this  section  also  contains  a  number  of  remarks  about 
which  methods  are  suited  to  obtaining  various  kinds  of  information. 

Explicit  definition  of  cognitive  task  analysis:  There  is  none.  Like  Cooke  (1994),  the  authors  place  cognitive  task 
analysis  far  down  in  their  taxonomy  of  methods,  identifying  it  with  a  small  set  of  methods,  notably  GOMS  analysis  and 
Mikyake's  constructive  interaction  method. 

Depth  of  description:  Relatively  detailed  compared  to  other  reviews,  most  detailed  for  machine-aided  methods  where 
examples  of  the  dialogues  between  machine  and  expert  are  given. 

Applications  of  interest  to  the  authors:  extraction  of  procedural  knowledge  for  the  purpose  of  building  ideal  student 
models  for  intelligent  tutoring  systems.  However,  the  review  is  much  broader  than  that;  only  the  concluding  section  is 
really  affected  by  that  purpose. 

Apparent  theoretical  perspective  of  the  authors:  Williams  is  a  cognitive  experimental  psychologist  by  training  who  has 
worked  primarily  on  military  training  applications  such  as  simulators.  He  seems  to  have  come  at  this  review  from  the 
perspective  of  Carnegie  Mellon  University-style  production  system  modeling  of  cognition. 

N.J.  Cooke  (1994).  Varieties  of  knowledge  elicitation  techniques.  International  Journal  of  Human-Computer  Studies, 
41,  801-849. 

This  review  covers  the  literature  up  until  1994. 

This  article  provides  an  extensive  review  of  some  70  knowledge  elicitation  techniques.  The  list  of  references  contains 
231  books  and  articles  on  the  subject.  Relatively  brief  reference  is  made  to  cognitive  task  analysis  approaches  or 
methodologies  (organized  sets  of  methods  or  techniques).  This  notwithstanding,  many  of  the  techniques  discussed  in 
this  article  appear  in  the  cognitive  task  analysis  methods  proposed  in  the  literature.  Moreover,  quite  a  few  of  the 
techniques  discussed  are  useful  for  practitioners  conducting  cognitive  task  analyses. 

The  author  views  knowledge  acquisition  as  a  modeling,  not  a  mining  activity.  The  major  problem  in  this  activity 
involves  the  abstracting  of  adequate  models  of  expert  knowledge  from  the  data  gathered  during  elicitation.  The  validity 
of  the  data  gathered  is  limited  because  much  knowledge  is  tacit  and  not  subject  to  conscious  introspection  and 
subsequent  verbalization.  The  validity  of  verbal  reports  has  therefore  been  subject  to  controversy.  Other  problems  in 
knowledge  acquisition  concern  the  unavailability  of  experts  and  the  communication  blocks  that  arise  when  experts  try 
to  convey  their  knowledge  to  non-experts  (in  an  attempt  to  satisfy  the  elicitor,  the  expert  communicates  whatever 
knowledge  is  accessible). 

No  explicit  definition  of  cognitive  task  analysis  is  provided.  On  p.813  cognitive  task  analysis  is  briefly  mentioned:  "(...) 
techniques  that  focus  more  intensely  on  cognitive  processes  (...)".  PARI  is  the  only  example  of  cognitive  task  analysis 
mentioned. 


20 


The  article  organizes  knowledge  elicitation  techniques  according  to  the  process  of  elicitation,  rather  than  to  elicitation 
stage  or  type  of  knowledge  elicited.  Three  families  of  techniques  are  distinguished:  (i)  observations  and  interviews,  (ii) 
process  tracing,  (iii)  conceptual  techniques. 

(i)  Observations  and  interviews  The  first  family  involves  rather  direct  methods  of  watching  experts  and  talking  with 
them.  These  methods  seem  well-suited  to  the  initial  phases  of  knowledge  elicitation,  although  the  results  are  often 
unwieldy  and  difficult  to  interpret. 

(ii)  Process  tracing  The  second  family  involves  techniques  that  are  generally  performed  concurrently  with  expert  task 
performance.  The  data  that  are  recorded  are  of  a  pre-specified  type  (verbal  reports,  eye  movements)  and  are  used  to 
make  inferences  about  the  cognitive  processes  or  knowledge  underlying  task  performance.  The  validity  of  the  data  is 
highly  dependent  upon  a  number  of  factors  that  should  be  carefully  considered  before  using  the  techniques,  e.g.  whether 
the  task  itself  involves  verbal  communication  or  has  high  cognitive  demands.  The  selection  of  cases  presented  to  the 
expert  to  solve  is  highly  critical:  the  cases  should  not  be  too  routine  (since  the  expert  uses  mainly  tacit,  unverbalizable, 
knowledge),  nor  too  unfamiliar  (since  the  expert  cannot  use  his  or  her  knowledge  and  the  result  will  not  be 
representative).  Process  tracing  techniques  result  in  large  and  often  unmanageable  data  sets  that  are  often  difficult  to 
interpret.  On  the  positive  side,  the  techniques  are  generally  carried  out  easily. 

(iii)  Conceptual  techniques  The  third  family  involves  indirect  techniques  that  produce  representations  of  domain 
concepts  and  their  structure  or  interrelations.  These  techniques  are  limited  in  the  types  of  knowledge  they  yield,  the 
focus  being  on  concepts  and  their  relations  rather  than  on  heuristics,  rules  and  strategies.  For  this  reason,  a  combination 
with  process  tracing  techniques  is  recommended.  In  addition,  the  analyst  requires  some  background  knowledge  in  order 
to  employ  these  techniques.  Experts  may  often  regard  these  techniques  with  suspicion,  due  to  their  artificiality.  On  the 
positive  side,  these  techniques  rely  less  on  verbal  reports,  they  are  more  suitable  for  dealing  with  data  from  multiple 
experts,  and  they  center  on  data  reduction. 

The  article  concludes  that  more  empirical  comparisons  between  techniques  are  required.  In  the  meantime,  it  advises  to 
employ  multiple  techniques  (with  the  caution  that  sequencing  of  techniques  may  influence  the  outcome  of  techniques), 
or  to  use  iterative  prototyping  and  evaluation  for  the  purpose  of  refining  the  knowledge  base. 

Depth  of  description:  Very  concise,  one-sentence  to  one-paragraph  summaries  with  pros  and  cons 
of  each  technique.  The  extensive  reference  list  should  guide  the  reader  to  more  elaborate  descriptions. 

Applications  of  interest  to  author:  Mostly  expert  systems,  although  the  techniques  discussed  can  be  applied  very 
broadly. 

Apparent  theoretical  perspective  of  author:  the  author  has  been  mainly  involved  with  the  development  of  conceptual 
techniques  (particularly,  the  Pathfinder  scaling  technique),  coming  from  an  experimental-psychological  background. 

B.  Crandall,  G.  Klein,  L.G.  Militello  &  S.P.  Wolf,  Tools  for  Applied  Cognitive  Task  Analysis.  Contract  Summary 
report  on  N66001-94-C-  7008.  Fairborn,  OH:  Klein  Associates,  July,  1994.  And  associated  draft  CD-ROM  training  in 
ACTA,  Applied  Cognitive  Task  Analysis,  1997. 

Literature  reviewed  extends  to  1993,  with  one  1994  reference. 

This  is  a  summary  report  on  a  Phase  I  Small  Business  Initiative  Research  (SBIR)  contract  intended  to  draw  upon  the 
experience  of  Klein  Associates  in  conducting  cognitive  task  analysis  in  order  to  produce  cognitive  task  analysis 
methods  that  could  be  used  by  training  developers  as  well  as  other  researchers.  The  work  of  the  project  included  a 
review  of  the  cognitive  task  analysis  literature  as  well  as  a  compilation  of  Klein  Associates'  in-house  experience  in 
conducting  cognitive  task  analyses.  Other  professionals  in  the  area  were  also  interviewed.  (The  report  contains  an 
annotated  bibliography  of  summaries  similar  to  this,  as  well  an  appendix  of  reports  of  the  interviews  of  other  cognitive 
task  analysis  experts.)  On  this  basis,  a  generic  applied  cognitive  task  analysis  (ACTA)  was  developed.  The  methods 
reviewed  included:  unstructured  interviews,  FAST  (functional  analysis  system  technique),  concept  maps,  conceptual 
graphs,  cognitive  interviews,  critical  decision  method,  constructed  scenarios,  repertory  grid,  modeling  methods,  and 
experimental  methods.  The  authors  report  that  concept  maps  are  quite  easy  to  do  but  can  become  excessively  complex. 
Klein  Associates  uses  concept  maps  in  initial  interviews,  developing  one  for  the  task,  one  for  the  equipment  and  one  for 
the  teamwork.  They  report  that  more  formalized  conceptual  graph  analysis  requires  more  time  and  specialized  training 
to  conduct.  Although  Klein  Associates  favors  the  use  of  the  critical  decision  method,  they  find  that  conducting  the 
necessary  interviews  requires  at  least  a  six  month  apprenticeship  and  that  some  experts  also  have  difficulty  participating 
in  this  method.  They  view  cognitive  modeling  methods  such  as  COGNET  and  GOMS  as  being  extremely  time- 
consuming  and  mostly  applicable  to  tasks  that  involve  interacting  with  a  keyboard.  They  view  the  use  of  experimental 
psychology  tasks  (like  scaling  of  similarity)  as  tedious  and  inappropriate  for  applied  projects.  Of  the  possible  ways  of 
representing  extracted  knowledge,  the  authors  prefer  the  "decision  requirements  table"  that  they  have  developed  and  the 
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"critical  cue  inventory".  They  have  not  found  concept  map  representations  or  other  methods  that  focus  on  extracting 
declarative  knowledge  to  be  very  useful.  Instead,  the  cognitive  literature  on  the  nature  of  expertise  has  proved  more 
useful.  Within  the  critical  decision  method,  they  like  to  seek  this  information:  1)  expert-novice  differences  in  dealing 
with  a  situation,  as  seen  by  the  expert;  2)  decision  points;  3)  goals  vs.  non-goals;  4)  critical  cues,  patterns,  and 
relationships;  5)  response  to  hypothetical  including  Marvin  Cohen's  crystal  ball  method;  and  6)  triggers.  With  respect 
to  goals,  they  look  at  how  goals  get  defined,  adopted  and  modified,  or  not  adopted  as  an  important  part  of  the  process. 
Critical  cues  refer  to  the  perceptual  aspects  of  tasks.  Triggers  refer  to  violations  of  expectations,  of  either  type.  The 
report  discusses  difficulties  some  people  have  had  in  developing  the  necessary  interview  skills  for  the  methods  that 
Klein  Associates  and  others  typically  use.  In  this  project,  they  aimed  to  develop  simplified  methods  that  Educational 
Specialists  can  use  to  interview,  observe,  and  help  Subject-Matter  Experts  articulate  the  basis  of  their  skills.  They 
wanted  to  develop  of  method  that  would  get  at  aspects  of  expertise,  mental  models  and  judgment  and  decision  making. 
For  this  purpose,  they  developed  corresponding  knowledge  elicitation  methods:  knowledge  audit  interviews,  schematics 
of  equipment,  tasks  and  team/organization,  and  scenarios  that  probe  key  decisions  and  associated  cues.  They  also 
developed  corresponding  knowledge  analysis  and  representation  methods:  1)  inventory  of  expertise  and  compilation  of 
specific  examples;  2)  aggregate  schematics  and  task  function  clusters;  and  3)  the  decision  requirements  table.  They 
estimate  that  interviewing  and  analyzing  4-5  SMEs  would  require  about  a  week's  work  by  educational  specialists  and 
assert  that  some  have  reacted  to  this  approach  as  potentially  reducing  the  work  typically  involved  in  a  course 
development  effort.  The  Knowledge  Audit,  for  example,  is  a  structured  interview  with  suggested  probe  questions  tied  to 
aspects  of  expertise  and  is  to  include  the  collection  of  examples  and  incidents  -  "sea  stories".  These  will  later  be  used 
for  instructional  materials.  The  schematics  are  intended  to  be  relatively  simple.  For  example,  the  team  schematic 
involves  identifying  the  significant  team  members  that  the  SME  interacts  with,  the  type  of  information  exchanged  with 
them,  and  the  criticality  of  that  information.  These  schematics  are  potentially  useful  as  advance  organizers  for  sections 
of  instruction.  To  look  at  judgment  and  decision  making,  SME's  are  presented  with  challenging  scenarios  and  asked 
about  the  embedded  judgments  and  decision  points.  This  is  an  easy  method  to  learn  and  scenarios  are  frequently 
available  already  in  military  training  contexts.  For  analysis,  the  decisions  involved  in  the  task  are  identified.  SMEs  are 
probed  about  the  information,  cues  and  other  factors  important  in  making  the  decision.  They  are  asked  which  decisions 
are  most  difficult.  This  information  can  help  to  set  training  objectives,  and  the  information  about  what  experts  do  can 
serve  as  models  for  trainees. 

Explicit  definition  of  cognitive  task  analysis:  Oddly,  there  does  not  seem  to  be  one.  In  the  end,  they  seem  to  view  it  as 
methods  for  identifying  cognitive  task  requirements. 

Depth  of  description:  The  report  is  not  very  well  organized  and  it  is  not  always  easy  to  figure  out  how  the  recommended 
methods,  let  alone  others  that  are  briefly  discussed,  would  be  executed.  However,  a  computer-based  training  for  the 
ACTA  process  has  now  been  developed.  It  appears  to  be  aimed  at  persons  revising  existing  courses  to  improve  their 
cognitive  content  and  it  is  quite  good  on  how  to  use  the  products  of  the  task  analysis  in  developing  improved 
instruction. 

Applications  of  interest  to  the  authors:  the  target  application  is  training  development  and  there  is  a  tendency  to 
emphasize  decision  making  tasks.  The  authors  say  that  cognitive  task  analysis  is  appropriate  only  for  tasks  that  involve 
true  expertise  taking  many  years  to  develop. 

Apparent  perspective  of  the  authors:  Klein  is  the  primary  proponent  of  the  naturalistic  decision-making  approach. 

P.J.M.D.  Essens,  J.J.  Fallesen,  C.A.  McCann,  J.  Cannon-Bowers,  &  G.  Dorfel.  (1994)  COADE:  A  Framework  for 
Cognitive  Analysis,  Design  and  Evaluation.  Final  Report  of  RSG-19  (Decision  Aids  in  Command  and  Control),  Panel  8 
(Defense  Applications  of  Human  and  Bio-  Medical  Sciences),  NATO  Defense  Research  Group.  (206  pages) 

Dates  of  references  suggest  coverage  up  to  about  1993,  with  a  heavy  representation  of  1990  and  1991. 

This  report  is  concerned  with  the  broader  topics  of  system  design  and  evaluation  within  the  application  area  of  decision 
support  for  military  command  and  control.  Only  a  portion  of  the  report  is  devoted  to  a  review  of  cognitive  task  analysis 
techniques.  Cognitive  task  analysis  is  covered  in  sections  5.1.3  (p.  43-45)  and  6.3  (p.  113-129).  The  report  assumes  that 
cognitive  task  analysis  will  be  done  in  the  context  of  prior,  more  traditional  behavioral  task  analyses.  Relatively  brief 
reference  is  made  to  cognitive  task  analysis  approaches  or  methodologies  (organized  sets  of  methods  and  techniques) 
that  have  been  proposed  by  other  workers.  More  attention  is  giving  to  reviewing  a  large  number  (about  35)  of  specific 
knowledge  elicitation  techniques  that  are  grouped  according  to  whether  they  yield  information  about  declarative 
knowledge  (e.g.  semantic  nets),  procedural  knowledge,  or  strategic  knowledge  (defined  as  knowing  when  and  how  to 
apply  the  other  kinds  of  knowledge,  as  well  as  metacognitive  or  monitoring  capabilities.)  Useful  concise  tables  briefly 
describe  these  techniques,  the  type  of  knowledge  representation  they  assume,  their  strength  and  weaknesses,  whether 
they  are  direct,  explicit  approaches  or  indirect  approaches  to  knowledge  extraction,  and  provide  references  to 
publications  discussing  the  techniques. 
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Explicit  definition  of  cognitive  task  analysis  (p.  1 13):  Cognitive  task  analysis  seeks  to  describe,  in  cognitive  terms,  how 
goals  and  tasks  are  accomplished.  The  value  of  cognitive  task  analysis  is  that  it  focuses  on  identification  of  the 
cognitive  basis  of  behavior  and  its  limitations  or,  positively  stated,  the  opportunities  for  support.  Cognitive  task  analysis 
is  concerned  with:  a)  the  knowledge  required  for  the  task  and  the  relationships  among,  and  organization  of  important 
concepts;  b)  the  mental  operations  for  retrieval,  storage,  transformation,  integration,  and  modeling  of  information;  c)  the 
meta-cognitive  processes  that  control  cognitive  effort  and  attention;  d)  cognitive  skill  development  and  progression  of 
knowledge  structures  from  novice  to  expert. 

Depth  of  description:  Very  concise.  More  informative  about  elicitation  techniques  than  about  entire  approaches  that  are 
alternatives  to  the  one  developed  in  the  report.  A  reasonably  useful  guide  to  further  reading,  with  some  sense  of 
evaluative  opinions. 

Applications  of  interest  to  authors:  As  noted  above,  the  design  of  decision  aids  for  military  command  and  control. 
However,  the  review  of  cognitive  task  analysis  techniques  is  quite  general,  not  much  affected  by  that  particular 
application  interest. 

Apparent  theoretical  perspective  of  authors:  Seems  to  be  transitional  between  behavioral  perspectives  and  more  modem 
cognitive  perspectives.  There  are  repeated  references  to  "cognitive  behavior"  and  goal  structures  are  assigned  to  an 
external  "behavioral  model"  of  the  task  rather  than  to  internal  cognitive  processes. 

B.E.  John  &  D.E.  Kieras  (1994).  The  GOMS  family  of  analysis  techniques:  Tools  for  design  and  evaluation  (Report 
CMU-CS-94-181).  Pittsburgh,  PA:  Carnegie  Mellon  University,  School  of  Computer  Science. 

The  references  in  this  technical  report  are  concentrated  around  the  late  eighties  and  the  early  nineties,  up  until  1994. 

This  paper  synthesizes  the  previous  work  on  GOMS  to  provide  an  integrated  view  of  GOMS  models  and  how  they  can 
be  used  in  design.  The  major  variants  of  GOMS  that  have  matured  sufficiently  to  be  used  in  real-world  design  and 
evaluation  situations  are  described  and  related  to  the  original  GOMS  proposal  and  to  each  other.  A  single  example  (text 
editing)  is  used  to  illustrate  all  of  the  techniques.  Guidance  is  provided  to  practitioners  who  wish  to  use  GOMS  for  their 
design  and  evaluation  problems,  and  examples  of  actual  applications  of  GOMS  techniques  are  presented. 

Insofar  as  this  paper  constitutes  a  review,  it  is  a  review  solely  of  GOMS  models.  However,  since  the  GOMS  family  is 
one  of  the  most  influential  types  of  cognitive  task  analysis  in  the  Human-Computer  Interaction  field,  it  is  worthwhile  to 
review  this  paper  here.  GOMS  models  are  best  viewed  as  engineering  models  of  human-computer  interaction,  in 
contrast  with  research-oriented  cognitive  models.  First,  these  models  must  be  able  to  make  predictions  early  in  the 
design  process  when  actual  data  cannot  yet  be  obtained  from  users.  Second,  the  models  need  to  be  used  by  computer 
system  designers  not  trained  in  psychology  or  human  factors  (the  authors  claim  that  most  GOMS  techniques  can  be 
taught  in  single  class  sessions  up  to  a  full-day  tutorial).  This  brings  with  it  the  need  for  explicit  procedures  for  use  and 
'canned'  psychological  knowledge  in  the  form  of  tables  of  parameter  values.  Third,  engineering  models  must  address  a 
useful  range  of  design  issues.  GOMS  models  cover  the  procedural  aspects  of  a  user  interface  well,  together  with 
lower-level  perceptual-motor  issues.  Fourth,  engineering  models  for  Human-Computer  Interaction  must  be  approximate 
in  nature:  with  20%  of  the  effort,  80%  accuracy  should  be  obtained.  This  is  what  GOMS  models  do. 

The  authors  provide  very  useful  background  information  on  the  concepts  used  by  GOMS  (e.g.,  the  difference  between 
goals  and  operators)  and  the  conceptual  framework  underlying  GOMS  (being  representative  of  stage  models  of  human 
information  processing).  It  is  made  clear  that  GOMS  is  a  form  of  task  analysis  that  describes  the  procedural, 
"how-to-do-it"  knowledge  involved  in  a  task.  GOMS  does  not  deal  with  declarative  knowledge  (e.g.,  device  knowledge) 
or  analogical  reasoning.  GOMS  is  a  form  of  task  analysis  that  only  deals  with  routine  operating  procedures,  not  with 
problem  solving  and  exploration.  However,  since  most  tasks  have  some  elements  of  routine  cognitive  skill,  GOMS 
techniques  may  be  used  widely.  An  important  aspect  of  using  GOMS,  which  sets  it  apart  from  other  task  analysis 
techniques,  is  that  the  analyst  must  start  with  a  list  of  top-level  tasks  or  user  goals.  GOMS  does  not  provide  this  list, 
other  task  analysis  techniques  (interviews,  observations,  intuition)  should  provide  it  (the  authors  do  not  consider  the 
opinion  that  a  list  with  top-level  tasks  is  more  important  and  more  difficult  to  get  than  carrying  out  a  GOMS  analysis 
once  you  have  defined  the  goals). 

The  paper  attempts  to  classify  the  various  GOMS  techniques  in  terms  of  task  type  (user  control  versus  system  control; 
sequential  versus  parallel)  and  the  type  of  design  information  required  (e.g.,  does  the  system  provide  a  method  for  every 
user  goal;  are  the  same  methods  used  throughout  for  the  same  goals;  what  sequence  of  overt  physical  operators  will  a 
user  perform;  what  will  be  the  learning  time  and  execution  time).  Using  this  classification  scheme,  analysts  can  choose 
a  particular  GOMS  technique. 
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Definition  of  cognitive  task  analysis:  No  explicit  definition  is  provided,  but  the  general  GOMS  concept  comes  close  to 
such  a  definition  (p.4):  "It  is  useful  to  analyze  knowledge  of  how  to  do  a  task  in  terms  of  the  components  of  goals, 
operators,  methods,  and  selection  rales". 

Depth  of  description:  Different  versions  of  GOMS  analysis  are  discussed  in  detail,  illustrated  with  examples  and  the 
relative  advantages  and  disadvantages  mentioned.  However,  the  level  of  detail  is  not  sufficient  for  novice  analysts  who 
want  to  learn  how  to  use  GOMS. 

Applications  of  interest  to  the  authors:  mostly  Human-Computer  Interaction  type  applications.  Eleven  cases  in  which 
GOMS  techniques  were  applied  are  discussed  in  some  detail.  Perhaps  surprisingly,  these  cases  are  not  restricted  to  the 
typical  word  processing  task  in  an  office  environment.  GOMS  techniques  have  been  applied  to  CAD  systems,  on-line 
databases  of  telephone  numbers,  bank  deposit  systems,  a  command  and  control  system  for  space  operations,  interfaces 
for  complex  televisions,  an  intelligent  associate  for  nuclear  power  plant  operators,  and  an  industrial  scheduling  system. 
In  almost  all  cases,  future  or  proposed  interfaces  are  evaluated  before  they  are  actually  implemented.  GOMS  techniques 
are  useful  in  those  cases  where  too  few  domain  experts  are  available  to  perform  user  tests.  GOMS  techniques  may  also 
supplement  field  tests  by  explaining  the  results  of  these  tests. 

Apparent  theoretical  perspective  of  the  authors:  Carnegie  Mellon  University-style  cognitive  modeling. 

E.J.H.M.  Merkelbach  &  J.M.C.  Schraagen  (1994).  A  framework  for  the  analysis  of  cognitive  tasks  (Report  TNO-TM 
1994  B-13).  Soesterberg,  The  Netherlands:  TNO  Human  Factors  Research  Institute. 

This  review  covers  the  literature  up  until  1993. 

The  report  is  based  on  a  literature  search  which  yielded  20  usable  techniques  or  methodologies  for  cognitive  task 
analysis.  Based  on  the  literature  search,  a  framework  is  presented  which  integrates  three  views  on  cognitive  task 
analysis:  task  modeling,  knowledge  modeling  and  cognitive  modeling.  The  task  modeling  view  concentrates  on  the 
decomposition  of  tasks  into  goals  and  subgoals.  Hierarchical  Task  Analysis  is  discussed  in  some  detail  as  an  example  of 
this  view.  The  knowledge  modeling  view  concentrates  on  the  knowledge  and  strategies  that  are  required  in  order  to 
accomplish  the  task's  goals.  The  KADS  methodology  for  knowledge-based  system  development  is  discussed  as  an 
example  of  this  view.  The  cognitive  modeling  view  concentrates  on  the  knowledge  and  strategies  actually  used  in  order 
to  accomplish  the  task's  goals.  The  GOMS  method  is  discussed  as  an  example  of  this  view.  Explicit  definition  of 
cognitive  task  analysis  (p.  24):  the  authors  make  the  strong  claim  that  only  when  the  three  views  mentioned  above  are 
used  together,  one  can  speak  of  'cognitive  task  analysis'.  Furthermore,  it  is  stated  that  the  task  environment  and  the 
knowledge  requirements  should  be  analyzed  first  before  the  task  performance  itself  is  studied  in  practice. 

Depth  of  description:  Of  the  20  techniques  or  methodologies  which  formed  the  input  for  the  framework,  only  HTA, 
KADS,  and  GOMS  are  discussed  in  detail,  with  evaluative  judgments.  Other  techniques  are  briefly  mentioned,  but  not 
discussed  in  detail.  All  20  analysis  techniques  are  scored  with  respect  to  the  emphasis  they  place  upon  the  three 
modeling  views. 

Applications  of  interest  to  authors:  The  report  proposes  a  speculative  connection  between  application  areas  and  the 
three  modeling  perspectives.  Task  modeling  is  viewed  as  important  for  describing  the  functional  structure  of  a  task 
domain  for  function  allocation  purposes.  Knowledge  modeling  is  viewed  as  important  for  developing  knowledge-based 
systems.  Cognitive  modeling  is  viewed  as  important  for  generating  interface  specifications,  for  assessing  and  improving 
workload,  for  designing  training  materials,  and  for  diagnosing  performance  and  predicting  performance  time. 

Apparent  theoretical  perspective  of  authors:  Eclectic,  with  inspiration  from  Carnegie  Mellon  University-like  cognitive 
modeling,  European  structured  approaches  to  knowledge-based  system  development  ('generic  task  method'),  and 
practical  task  analytic  behavioral  approaches. 

A.  Whitefield  &  B.  Hill  (1994)  Comparative  analysis  of  task  analysis  products.  Interacting  with  Computers,  6(3), 
289-309. 

References  extend  to  1992-93  but  do  not  seem  to  include  recent  developments  of  the  task  analysis  approaches  reviewed. 

By  "products"  this  paper  means  the  outcome  product  of  task  analysis  processes.  Four  task  analysis  processes  are 
reviewed  from  the  perspective  of  applications  to  human-computer  interaction;  Hierarchical  Task  Analysis;  Task 
Knowledge  Structures;  GOMS;  Cognitive  Task  Analysis/Interacting  Cognitive  Subsystems. 

The  authors  seem  to  be  concerned  primarily  with  tasks  of  low  cognitive  complexity  such  as  word  processing  or  other 
document  manipulation  tasks.  To  be  more  precise,  they  are  concerned  with  "interactive  work  systems(IWS)"  for  such 
tasks.  They  have  developed  a  schema  for  use  in  discussing  task  analysis  "products"  that  they  use  to  characterize  the 
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above  processes,  asserting  that  a  task  analysis  product  should  contain  descriptions  of  the  following  components:  IWS 
behaviors,  IWS  behavior  sequences,  task  goals,  a  decomposition,  work  domain  objects  [e.g.  documents]  and  their 
attributes,  abstract  IWS  structures,  physical  IWS  structures,  performance,  cross-references  between  these  components. 
The  authors  seem  to  be  somewhat  dissatisfied  with  all  of  the  task  analysis  processes  they  review  because  all  are  focused 
on  the  human  user  and  do  not  provide  independent  descriptions  of  the  work  objects  or  the  physical  characteristics  of  the 
IWS.  They  comment  that  this  can  make  it  difficult  to  ascertain  whether  alternative  workstation  designs  could  perform 
the  same  work.  They  believe  that  the  GOMS  approach  can  handle  only  serial  operations,  a  view  that  is  not  up  to  date 
with  current  versions  of  the  GOMS  approach.  They  characterize  Barnard's  approach  as  being  totally  concerned  with 
specifying  internal  cognitive  processes  of  the  user,  as  having  aspirations  to  "psychological  reality"  in  those  descriptions 
and  as  giving  a  description  only  at  that  base  level,  without  retaining  any  hierarchical  description.  This  approach  and 
GOMS  are  characterized  as  tending  to  a  fixed  vocabulary  of  low-level  operations  while  the  other  approaches  are  more 
free- form  and  therefore  perhaps  more  dependent  upon  the  specialized  expertise  of  the  analyst. 

This  paper  does  not  provide  a  definition  of  cognitive  task  analysis  and  does  not  appear  to  distinguish  cognitive  task 
analysis  from  task  analysis  in  general. 

Depth  of  description:  This  paper  makes  comments  about  the  various  methods  but  really  does  not  describe  them  at  a 
useful  level  of  detail;  the  emphasis  is  on  the  author's  own  descriptive  schema,  which  does  not  seem  to  be  very 
well-adapted  to  the  actual  characteristics  of  the  task  analysis  approaches  described.  It  is  useful  primarily  for  a  few 
comments  generated  from  the  authors'  perspective,  most  of  which  I  have  noted  above. 

Applications  of  interest  to  the  authors:  Apparently  Human-Computer  Interaction  design  for  systems  like  document 
processing  systems. 

Apparent  theoretical  perspective  of  authors:  The  authors  appear  to  be  writing  from  the  perspective  of  computer  systems 
engineers  or  software  designers. 

Dehoney,  J.  (1995).  Cognitive  task  analysis:  Implications  for  the  theory  and  practice  of  instructional  design. 
Proceedings  of  the  1995  Annual  National  Convention  of  the  Association  for  Educational  Communications  and 
Technology  (AECT)  Anaheim,  CA.  (ERIC  Document  Reproduction  Service  No.  ED  383  294). 

Dates  covered:  This  review  spans  the  period  from  1985  to  1992. 

Abstract:  Dehoney's  CTA  review  is  broken  up  into  3  sections:  (1)  Summary  of  the  underlying  theory  of  CTA,  with  a 
focus  on  problem-solving  and  expertise,  (2)  Review  of  CTA  models  and  techniques  (including  weaknesses),  and  (3) 
Description  of  the  implications  of  CTA  techniques  for  instructional  design.  The  state  of  current  instructional  design  (ID) 
task  analysis  results  in  a  linear  description  or  hierarchical  ordering  of  skills  required  to  achieve  the  task,  which  are  fine 
for  top  down,  or  bottom  up,  analysis.  Cognitive  psychologists  have  extended  such  analyses  into  methods  for  describing 
how  people  solve  difficult  problems,  with  the  most  important  skills  being  "metacognitive  strategies  for  selecting 
relevant  information,  prioritizing  and  revising  goals,  and  working  between  multiple  versions  of  the  problem 
representation."  The  underlying  theory  of  CTA  in  terms  of  problem-solving  involves  the  problem  representation  of 
experts  (or  novice-expert  comparisons).  Learners  internally  represent  objects,  actions  of  objects,  strategies  to  use  with 
those  objects,  and  any  constraints  of  those  objects,  actions,  and  strategies.  In  experts,  these  representations  are  linked  to 
the  specific  domain  knowledge.  The  process  of  solving  a  problem  is  working  through  the  problem  space  from  the  initial 
state  (i.e.,  givens  of  the  problem)  to  the  goal  state  (i.e.,  problem  solution).  Performance  by  experts  has  been 
characterized  with  several  consistent  features  (e.g.,  broader  and  better  organized  domain  knowledge,  better  able  to 
recognize  patterns,  focus  on  underlying  principles  instead  of  surface  features),  and  it  is  the  goal  of  CTA  to  extricate 
their  representations  and  strategies.  Several  researchers  describe  various  procedures  for  CTA  focusing  on  different 
aspects  of  experts'  problem  solving-for  instance,  deriving  an  implicit  rule  base  for  problem  solution  (Ohlsson,  1990),  or 
determining  expert  organization  of  domain  knowledge  into  schemata,  semantic  webs,  or  mental  representations  (Means, 
1993;  Lesgold,  Lajoie,  Logan,  &  Eggan,  1990;  Nelson,  1990).  However,  Gardner  (1985)  and  Roth  and  Woods  (1989) 
have  proposed  formal  CTA  models.  Gardner  (1985)  states  that  the  purpose  of  CTA  is  to  identify  performance 
components,  knowledge  structures,  and  metacognitive  knowledge  underlying  a  task.  Roth  and  Woods  (1989)  define 
CTA  as  a  two  part  process.  The  first  part  is  to  create  a  competence  model  (i.e.,  define  competent  performance  on  the 
task,  or  create  a  model  of  the  problem-solving  environment),  which  enables  the  analyst  to  identify  problem  areas.  The 
second  part  is  to  derive  a  performance  model  of  how  the  task  is  actually  achieved  in  practice,  whereby  domain-specific 
knowledge  is  elicited  through  interviewing  and  other  observational  techniques.  Dehoney  describes  these  two  models  as 
compatible,  with  Gardner's  viewed  as  part  of  a  performance  model  for  CTA  and  Roth  and  Woods'  as  a  broad 
competency  model.  Dehoney  grouped  CTA  strategies  for  conducting  CTA  into  three  general  categories:  (1)  methods  for 
using  domain  knowledge  to  structure  the  analysis  (i.e.,  review  existing  materials  on  the  task,  interview  practitioners, 
hold  discussions  with  panels  of  experts),  (2)  development  of  focus  problems  (i.e.,  to  gain  an  understanding  of  cognitive 
skills  required  for  task,  study/create/discuss  representative  or  critical  scenario  problem-solving  in  the  domain  by 
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experts),  and  (3)  knowledge  elicitation  (i.e.,  at  the  performance  level:  observing  and  acquiring  information  about  the 
task;  at  the  knowledge  structure  level:  interviews,  discussions,  protocol  analysis,  simulations;  at  the  metacognitive  skills 
level:  protocol  analysis,  reconstruction,  developing  mental  models).  Results  from  these  methods  are  then  organized  in  a 
manner  which  is  useful  for  task  performance  (e.g.,  tables  with  each  knowledge  type  and  examples  listed).  Although 
there  are  similarities  in  the  ID  and  CTA  procedures  (e.g.,  hierarchical  task  descriptions),  differences  in  terminology  and 
emphasis  make  it  difficult  (but  not  insurmountable)  for  the  instructional  designer.  Even  though  cognitive  psychologists 
may  not  find  any  benefit  from  ID  literature  (e.g.,  the  literature  already  shows  some  CTA-designed  instruction  which 
successfully  remediates  declarative  and  procedural  knowledge:  Sherlock),  instructional  designers  would  benefit  from 
current  CTA  methods  (e.g.,  by  addressing  metacognitive  skills  required  in  the  task,  by  providing  more  specific 
dimensions  to  the  current  task  analysis  methods).  Cognitive  psychologists  continue  to  identify,  research,  and 
instructionally  manipulate  complex  mental  models,  and  thus  will  continue  to  discover  worthwhile  instructional 
techniques  for  complex  problem-solving  domains.  Instructional  designers  also  need  to  capitalize  on  this  importance  of 
cognition. 

Definition  of  CTA:  Cognitive  task  analysis  models  and  techniques  are  useful  when  the  object  of  training  is  to  move 
people  to  the  level  of  expert  performance  on  complex  problem-solving  tasks.  Such  techniques  "focus  on  illuminating 
the  ’covert  heuristic'  (Wilson  and  Cole,  1990)  used  by  experts  to  solve  problems,  and  result  in  a  description  of  an 
expert's  mental  model  of  the  problem."  (p.  1 14) 

Depth  of  description:  The  focus  of  Dehoney's  paper  is  with  cognitive  task  analysis  of  problem-solving.  She  provides 
rich  descriptions  and  examples  of  elicitation  techniques  used  in  the  literature  (late  80s,  early  90s).  A  brief  summary  of 
the  roots  of  CTA  is  also  provided. 

Applications  of  interest  to  the  authors:  Dehoney  points  out  that  the  major  applications  currently  discussed  in  the 
literature  are:  air  traffic  controllers  (Means,  1993),  electronic  specialists  troubleshooting  avionics  equipment-testing 
workstations  (Lajoie  &  Lesgold,  1992;  Lesgold  &  Lajoie,  1991;  Lesgold,  Lajoie,  Logan,  &  Eggan,  1990;  Means,  1993), 
and  nuclear  power  plant  operators  (Roth  &  Woods,  1992;  Roth,  Woods,  &  Pople,  1992).  The  focus  of  CTA  is  in  terms 
of  problem-solving  and  how  experts  go  about  solving  problems,  as  compared  to  less  experienced  persons. 

Apparent  theoretical  perspective  of  the  author:  Cognitive  psychologist  perspective  with  strong  emphasis  on 
problem-solving. 

D.  DuBois  &  V.L.  Shalin  (1995)  Adapting  cognitive  methods  to  real-  world  objectives:  An  application  to  job 
knowledge  testing.  In:  P.D.  Nichols,  S.F.  Chipman  &  R.L.  Brennan  (Eds.)  Cognitively  Diagnostic  Assessment  (pp.  189- 
220).  Hillsdale,  NJ:  Lawrence  Erlbaum  Associates. 

Literature  cited  extends  to  1993. 

This  chapter  is  not  really  a  review  of  cognitive  task  analysis  methods  but  the  presentation  of  an  approach  to  cognitive 
task  analysis  with  the  purpose  of  improving  the  quality  of  job  knowledge  tests  by  including  questions  that  probe  types 
of  knowledge  revealed  by  cognitive  task  analysis.  It  also  presents  a  case  example  involving  Marines'  land  navigation 
skill.  The  approach  begins  with  a  larger  perspective  than  is  usual,  employing  relatively  standard  personnel  psychology 
methods  for  the  analysis  of  jobs  in  order  to  identify  the  tasks  that  are  worthy  of  detailed  analysis.  Current  test 
development  methods,  they  say,  conceptualize  knowledge  as  an  accumulation  of  facts,  concepts  and  skills  and  provide 
little  information  about  how  knowledge  is  organized  and  used  in  job  performance.  Existing  test  development  methods 
are  based  on  ratings  of  job  importance,  saliency,  or  frequency  of  use  and  do  not  serve  well  to  differentiate  novices  from 
expert  performers.  In  fact,  they  generate  approximately  the  same  results  whether  experts  or  novices  are  used  as 
informants.  The  methods  of  knowledge  elicitation  discussed  and  used  included  protocol  analysis,  coaching  of  a 
collaborator  by  the  expert,  and  analysis  of  naturally  occurring  team  communication,  as  well  as  supplementary  use  of 
retrospective  probe  questioning  and  critical  incident  methods.  This  chapter  gives  much  more  attention  to  the  resulting 
knowledge  representation  than  most:  the  knowledge  representation  organizes  most  of  the  task  analysis  and  test 
development  activity.  The  chapter  reviews  the  characteristics  of  knowledge  that  have  been  shown  to  differentiate  expert 
and  novice  knowledge:  ontology  (cf.  Chi,  Feltovich  &  Glaser,  1981:  differences  in  objects,  relations  and  attributes), 
explanatory  knowledge  (understanding  why  procedures  are  constructed  the  way  they  are  and  what  principles  must  be 
respected  in  modifying  them),  tacit  knowledge  (relevant  conditions  for  task  goals,  knowing  how  to  integrate  bits  of 
knowledge  and  information  into  performance),  and  recognition  of  goal  attainment  (which  involves  identifying  both  final 
states  and  -  -sometimes  —  whether  proper  process  was  used  in  getting  there.)  It  is  noted  that  representing  novice 
knowledge  may  require  the  addition  of  "buggy"  knowledge  or  misconceptions.  The  knowledge  representation  used  was 
a  plan(or  method)-goal  graph,  similar  to  a  GOMS  analysis.  The  nodes  of  this  graph  were  annotated  with  additional 
information:  1)  concepts  and  principles  explaining  why  the  method  works,  2)  procedure  selection  knowledge,  3)  pattern 
recognition  knowledge  of  contexts  required  to  select  or  execute  task  procedures,  (4)  procedure  execution  knowledge 
such  as  steps  or  subgoals,  (5)  goal  attainment  knowledge  —  including  priority,  sequencing  and  standards.  The  nodes 
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were  also  annotated  with  candidate  test  questions  and  descriptions  of  typical  mistakes.  In  conducting  the  task  analysis 
of  land  navigation  skill,  several  different  "experts"  of  varying  skill  were  used,  along  with  2  novices  and  2  "decayed 
experts"  who  had  not  actually  practiced  the  skill  recently.  Videotaped  think  aloud  protocols  were  supplemented  with 
retrospective  questioning  (How  did  you  know?  Why  did  you  select?)  to  fill  in  missing  information.  Analysis  of  team 
communication  revealed  the  existence  of  vocabulary  for  many  perceptual  aspects  of  the  task.  The  study  was  done  in  two 
different  terrains  and  retrospective  interviews  were  used  to  explore  the  effects  of  different  terrains  and  conditions.  The 
critical  incident  technique  was  used  primarily  to  identify  errors  and  buggy  knowledge.  The  authors  comment  that  they 
were  aiming  for  a  practical,  cost  effective  method  so  that  there  were  some  methodological  trade-off:  relatively  few 
protocols  were  collected  in  relation  to  the  procedures  involved  (65  identified),  the  analysis  were  not  taken  to  the  level  of 
detail  required  for  computational  modeling  and  no  experimental  analyses  of  expertise  were  done.  (In  other  words,  they 
did  not  use  most  of  the  techniques  for  knowledge  acquisition  described  in  most  reviews.)  For  the  purposes  of  test 
development,  the  method-goal  graph  information  was  converted  to  a  matrix  with  domain  specific  methods  as  rows  and 
the  types  of  annotated  knowledge  described  above  (e.g.  procedure  selection)  as  columns.  This  matrix  specified  the  types 
of  test  questions  to  be  developed.  In  addition,  expert  ratings  of  diagnosticity,  rather  than  importance  or  frequency  of 
use,  were  used  in  selecting  which  questions  to  use.  The  authors  also  note  that  the  method-goal  graph  representation  is 
useful  for  practical  purposes  in  demonstrating  that  one  is  testing  job-relevant  knowledge. 

Explicit  definition  of  cognitive  task  analysis:  None  is  given.  Implicitly,  their  definition  seems  to  be  the  application  of 
cognitive  science  methods  to  job  or  task  analysis. 

Depth  of  description:  There  is  not  much  detail  to  the  description  of  the  methods  used.  There  is  considerable  detail  about 
the  knowledge  representation  used  to  organize  their  activity  and  about  how  the  different  methods  were  combined  and 
adapted  in  their  case  example. 

Applications  of  interest  to  authors:  as  noted,  the  explicit  purpose  was  to  inform  the  design  and  development  of 
improved  job  knowledge  tests  that  would  better  correlate  with  measures  of  hands-  on  performance.  However,  the 
authors  also  recognize  that  their  task  analysis  approach  is  equally  applicable  to  training  needs  analysis  and  design.  The 
knowledge  representation  is  adapted  from  one  (Geddes,  1989;  Sewell  &  Geddes,  1990)  developed  for  Human- 
Computer  Interaction  design. 

Apparent  perspective  of  the  authors:  A  combination  of  personnel  psychology  (DuBois)  and  mainstream  cognitive 
science  (Shalin). 

Hall,  E.  P.,  Gott,  S.  P.  &  Pokomy,  R.  A.  (1995).  A  Procedural  Guide  to  Cognitive  Task  Analysis:  The  PARI 
Methodology.  AFMC,  Brooks  Air  Force  Base,  Texas,  USA.  (Technical  Report  No.  1995-0108). 

Dates  covered:  Even  though  this  technical  paper  just  recently  came  out,  the  four  general  CTA  procedures  reviewed 
therein  span  the  80s.  The  PARI  methodology  itself  arose  during  this  time  and  evaluations  of  the  technique  and  resulting 
curricula/tutors  took  place  in  the  mid-  to  late  80s. 

Abstract:  Developing  effective  instruction  for  complex  problems-solving  tasks  requires  analysis  of  the  cognitive 
processes  and  structures  that  contribute  to  task  performance.  This  report  describes  in  great  detail  the  exhaustive  data 
collection  procedures  associated  with  a  CTA  technique  known  as  the  PARI  methodology  (for:  precursor,  action,  result, 
and  interpretation).  The  method  was  developed  at  Brooks  AF  Base  (Basic  Job  Skills  program,  originally  headed  by 
Sherrie  Gott,  now  led  by  Ellen  Hall).  The  PARI  method  represents  one  component  of  an  integrated  technology  for 
developing  and  delivering  training  of  cognitively  complex  tasks.  The  specific  data  collection  procedures  they  discuss  in 
this  paper  can  be  considered  an  extension  of  existing  task  analysis  techniques  and  are  based  on  studies  of  over  200  AF 
technicians  in  aircraft  maintenance  specialties  whose  primary  task  is  troubleshooting.  The  procedures  derived  from 
these  studies  impose  a  structure  on  the  knowledge  acquisition  task  which  captures  the  cognitive  as  well  as  the 
behavioral  components  of  troubleshooting  skill.  The  structured  interview  approach  yields  data  that  allow  qualitative 
comparisons  of  problem-solving  performances  within  and  across  technical  skill  levels  (novices,  experts,  shades  of  gray 
in  between).  Such  analyses  have  informed  instruction  developed  under  the  program  by  revealing  the  developmental 
course  of  skill  acquisition  and  the  components  of  expertise  which  are  the  training  targets.  That's  a  definite  plus.  More 
recent  analyses  have  identified  skill  and  knowledge  overlaps  across  maintenance  specialties  and  are  informing  training 
designed  to  facilitate  knowledge  transfer.  A  future  goal  of  the  program  is  to  examine  the  generality  of  the  PARI 
methodology  and  the  extent  to  which  it  can  be  applied  to  problem-solving  tasks  in  non-maintenance  domains.  The  claim 
of  the  authors  is  that  this  PARI  method  represents  a  "general  purpose"  tool,  but  it's  currently  quite  bound  to  eliciting  and 
structuring  troubleshooting  knowledge  &  skills. 

Definition  of  CTA:  Cognitive  task  analysis  is  defined  in  contrast  to  behavioral  task  analysis,  differing  along  an 
instruction  dimension.  That  is,  behavioral-rational  task  analysis  specifies  component  steps  of  observable  behaviors  as 
targets  of  instruction,  while  CTA  gets  at  the  underlying  psychological  processes  and  knowledge  structures  that  are 
required  to  produce  the  correct  overt  behaviors  at  the  appropriate  time  (p.  12).  The  primary  focus  of  behavioral  task 
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analysis  (BTA)  is  procedural  knowledge,  while  CTA  additional  elicits  and  structures  declarative  knowledge.  [Note:  I've 
personally  asked  various  PARI  persons  to  categorize  "strategic  knowledge"  (i.e.,  whether  it's  more  declarative  or 
procedural  in  representation  and  application).  But  unfortunately,  there's  been  no  agreed-upon  answer  yet  to  my  query.] 

Depth  of  description:  The  focus  of  this  paper  was  primarily  on  the  PARI  methodology.  These  sections  were  written  in 
great  depth.  However,  the  "review"  of  the  four  alternative  task  analysis  approaches  were  summarized  in  only  moderate 
detail.  lire  point  of  the  summaries  was  to  show  how  various  parts  of  each  system  have  been  incorporated  into  the  PARI 
approach.  The  four  alternative  methods  include:  (a)  Instructional  Systems  Design,  (b)  GOMS,  (c)  Knowledge 
Engineering,  and  (d)  Verbal  Protocol  Analysis. 

Applications  of  interest  to  the  authors.  The  primary  focus  of  the  PARI  methodology  has  been  to  capture  the  cognitive 
aspects  of  certain  military  avionics  jobs.  Specifically,  the  goal  has  been  to  obtain  rales  underlying  troubleshooting  skills 
and  develop  the  associated  assessment  items  that  reflect  system,  procedural,  and  strategic  knowledge  types. 

Apparent  theoretical  perspective  of  the  author:  The  theoretical  basis  of  the  PARI  approach  represents  the  integration  of 
parts  of  the  four  alternative  approaches  summarized  in  the  report  (i.e.,  ISD,  GOMS,  protocol  analysis,  knowledge 
engineering).  In  other  words,  the  authors  write  from  a  basic  cognitive  science  approach  along  with  cognitive 
psychology  (i.e.,  individual  differences  approach  with  regard  to  novice-expert  elicitations). 

R. R.  Hoffman,  N.R.  Shadbolt,  A.M.  Burton,  &  G.  Klein  (1995)  Eliciting  knowledge  from  experts:  A  methodological 
analysis.  Organizational  Behavior  and  Human  Decision  Processes,  62(2),  129-  158. 

This  paper  has  a  very  extensive  reference  list  with  heavy  coverage  of  the  late  1980's  and  some  through  1992  and  later. 
This  paper  is  a  very  generalized  review  of  approaches  for  capturing  expert  knowledge  for  purposes  ranging  from 
reduction  of  performance  error  through  conventional  training  to  the  creation  of  artificial  expert  systems.  The  methods 
considered  range  from  the  use  of  texts,  reports  and  courses  (which  they  call  indirect)  to  the  analysis  of  experts 
performing  naturalistic  tasks,  various  types  of  interviews  and  various  types  of  "contrived"  techniques.  It  is  relatively 
stronger  on  the  less  formal  methods  such  as  interviews  than  on  the  "contrived"  techniques,  which  are  mostly 
experimental  methods  from  cognitive  psychology.  It  discusses  degrees  and  kinds  of  expertise  and,  briefly,  ways  of 
identifying  experts  to  use  in  knowledge  elicitation,  as  well  as  several  different  approaches  for  finding  case  examples  to 
be  used  in  studying  experts'  problem  solving  behavior.  The  value  of  structured  (vs.  totally  unstructured)  interviews  is 
discussed  and  references  on  this  subject  are  given,  along  with  a  detailed  example  of  generic  probe  questions.  The 
"contrived  techniques"  discussed  include  some  unusual  ones  like  decision  analysis  and  the  use  of  group  decision¬ 
making  situations  as  well  as  the  more  usual  rating  and  sorting  tasks,  repertory  grid,  and  explicit  construction  of 
conceptual  graphs.  The  use  of  experimentally  constrained  problem  solving  conditions,  in  which  time  or  information  is 
abnormally  constrained,  is  also  mentioned.  There  is  little  detail  on  these  techniques,  although  there  is  some  mention  of 
and  references  to  computerized  knowledge  acquisition  tools  that  implement  such  techniques.  The  latter  portion  of  the 
paper  discusses  experience  with  knowledge  elicitation  approaches  and  some  comparative  evidence.  It  is  admitted  that 
contrived  tasks  seem  to  be  more  efficient  providers  of  information  than  interviews.  Some  possible  problems  are 
discussed:  apparently  inaccessible  expert  knowledge,  interactions  between  the  method  of  elicitation  and  the  knowledge 
or  strategies  that  are  evoked  in  the  expert,  and  biases  that  are  introduced  by  the  structure  of  the  interview  or  contrived 
task.  Towards  the  end  of  the  paper  there  is  a  suggestion  of  an  overall  approach:  study  of  documentary  sources  or 
courses  to  familiarize  the  analyst  with  the  concepts  and  vocabulary  of  the  domain,  followed  by  unstructured  interviews, 
expert  review  of  the  resulting  "first  pass  knowledge  base",  followed  by  structured  interviews,  think  aloud  problem 
solving  and  contrived  tasks  for  refinement.  Throughout,  some  useful  comments  on  the  social  niceties  of  working  with 
experts  are  made.  At  the  end,  the  possibility  of  a  taxonomy  of  tasks  that  could  be  associated  with  recommended 
knowledge  elicitation  techniques  is  raised. 

Definition  of  cognitive  task  analysis:  the  term  is  not  prominent  in  this  paper  and  no  definition  is  given. 

Depth  of  description:  Although  the  paper  is  quite  wordy  and  discursive,  most  techniques  are  not  described  in  useful 
detail,  with  the  exception  of  generally  applicable  probe  questions  for  a  semi-structured  interview. 

Applications  of  interest  to  the  authors:  as  noted  above,  these  are  wide-ranging  but  mention  of  the  development  of  expert 
systems  may  be  over-represented. 

Apparent  perspective  of  the  authors:  they  show  more  interest  in  techniques  that  are  typical  of  social  or  organizational 
psychologists  as  compared  to  experimental  psychology  techniques,  although  in  the  end  they  recommend  a  mixture  of 
methods. 

S. E.  Gordon  &  R.T.  Gill  (1997).  Cognitive  task  analysis.  In  C.E.  Zsambok  &  G.  Klein  (Eds.),  Naturalistic  Decision 
Making  (pp.  131-140).  Mahwah,  NJ:  Lawrence  Erlbaum  Associates. 
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This  paper  covers  the  literature  up  until  1994,  with  a  heavy  emphasis  on  the  late  80s  and  early  90s. 

This  short  chapter's  origins  lie  in  a  panel  discussion  that  was  held  in  1994  during  the  second  Naturalistic  Decision 
Making  Conference  held  in  Dayton,  Ohio.  Unfortunately,  the  panel  members  are  not  mentioned  in  the  chapter.  They 
were:  Hoffman,  Kaempf,  Gott,  Gordon,  Ryder,  and  McNeese. 

The  chapter  contrasts  cognitive  task  analysis  (CTA)  with  traditional  behavioral  task  analysis  methods.  Because  many 
jobs  are  beginning  to  involve  complex  cognitive  processing,  CTA  methods  are  better  suited  to  modem-day  jobs  than 
behavioral  task  analysis  methods.  Tasks  that  are  cognitively  complex  are  well-suited  for  CTA  (Gordon  [1992]  claims  to 
have  developed  a  software  tool  that  helps  an  analyst  estimate  the  cognitive  complexity  of  a  task).  In  practice,  many 
tasks  contain  both  cognitively  complex  and  more  behavioral  subtasks  at  the  same  time,  such  that  a  variety  of  task 
analysis  methods  is  most  useful.  It  is  also  recommended  to  use  a  variety  of  CTA  methods,  because  each  method  may 
get  at  different  types  of  knowledge  or  is  biased  in  different  ways. 

The  chapter  discusses  five  CTA  methods  in  some  detail:  Concept  Mapping  and  Expert  Design  Storyboarding; 
COGNET;  Conceptual  Graph  Analysis;  PARI;  Critical  Incidents,  Tough  Cases,  and  Constrained/Contrived  Cases. 
Overall  methodologies  are  not  discussed.  The  discussion  of  these  methods  is  fairly  shallow  and  non-evaluative.  The 
authors  note  that  these  methods  differ  along  a  number  of  dimensions,  such  as  how,  and  by  whom,  scenarios  or  test  cases 
are  generated,  the  type  of  cases  analyzed,  and  the  types  of  representational  formalisms  used. 

The  chapter  concludes  with  issues  for  the  future.  One  is  the  issue  of  when  to  use  what  method.  The  authors  propose  that 
system  designers  evaluate  the  type  of  information  required  for  a  particular  project  first,  and  then  select  a  CTA  method 
that  yields  that  type  of  information  (the  authors  do  not  specify  what  they  mean  by  'type  of  information',  so  this  advice  is 
not  really  helpful).  A  second  issue  concerns  the  validity  and  reliability  of  various  CTA  methods.  Validity  concerns  the 
question  whether  the  domain  expertise  is  adequately  captured.  Since  the  authors  view  CTA  as  a  modeling  enterprise,  the 
model  of  expert  performance  can  be  used  to  generate  predictions  and  these  can  be  tested  empirically.  A  second  way  to 
validate  the  model  is  to  look  at  the  success  of  the  artifact  it  has  supported  (e.g.,  the  training  program  designed  with  the 
aid  of  the  model).  Unfortunately,  the  artifact  itself  may  be  a  confounding  factor  (e.g.,  the  training  program  could  have 
led  to  improved  results  without  the  CTA  being  conducted  in  advance).  Reliability  of  CTA  methods  concerns  the  issue 
whether  different  analysts  using  the  same  method  end  up  with  the  same  results  or  not.  The  authors  note  that  there  is  a 
scarcity  of  publications,  seminars  and  courses  on  how  to  conduct  a  CTA.  This  may  lead  to  low  reliability  of  methods. 
Relatively  simple  methods  that  can  be  taught  to  training  and  system  analysts  should  be  identified.  Finally,  an  issue  for 
the  future  is  the  translation  of  the  results  of  CTA  into  a  final  system,  training  program  or  interface.  There  is  still  a  gap 
here  that  may  have  more  ramifications  than  the  choice  of  any  particular  CTA  method. 

Definition  of  cognitive  task  analysis:  CTA  is  fairly  narrowly  defined  as  "a  theory  or  model  of  how  experts  in  a  field 
perform  their  tasks"  (p.132).  CTA  attempts  to  "identify  how  (primarily)  experts  perform  a  cognitive  task"  (p.133).  More 
specifically,  CTA  methods  "capture  the  richness  of  the  knowledge  base  and  cognitive  processing  embodied  in  expert 
performance  of  a  task"  (p.132). 

Depth  of  description:  concise,  half-page  descriptions  of  five  methods.  The  methods  are  compared  along  some 
dimensions,  but  this  is  not  very  enlightening. 

Applications  of  interest  to  authors:  very  broad,  no  specific  application  mentioned  in  any  detail  (CTA  is  said  to  be  useful 
for  developing  system  interfaces,  intelligent  support  or  decision  aids,  and  training  programs).  The  focus  is  more  on 
methods  than  on  applications.  No  attempt  is  made  to  link  methods  to  applications. 

Apparent  theoretical  perspective  of  the  authors:  general  academic  cognitive  psychological  perspective,  with  a  focus  on 
semantic  networks  as  representational  formalisms  for  domain  knowledge. 
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CHAPTER  IV 


Report  on  the  NATO-ONR  Workshop  on  Cognitive  Task  Analysis 
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3769  ZG  Soesterberg,  The  Netherlands 
Tel:  [31]  (346)  356323  -  Fax:  [31]  (346)  353977 
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SUMMARY 

NATO  Research  Study  Group  27  on  Cognitive  Task  Analysis  has  organized  a  workshop  with  experts  in  the  field  of 
cognitive  task  analysis.  The  Workshop  was  held  in  Washington,  D.C.,  USA,  October  30-November  1,  1997.  The  goal  of 
the  workshop  was  to  assess  the  state  of  the  art  by  bringing  together  a  diverse,  yet  representative  sample  of  experts  in  the 
field.  The  outcome  of  the  workshop  will  be  an  edited  book  and  a  report  to  NATO. 

This  chapter  briefly  describes  the  organizational  and  financial  structure  of  the  workshop.  A  summary  of  all 
presentations  is  provided.  The  chapter  ends  with  a  list  of  important  issues  that  arose  from  the  discussions  in  the 
workshop. 

1  INTRODUCTION 

NATO  Research  Study  Group  27  on  Cognitive  Task  Analysis  has  organized  a  workshop  with  experts  in  the  field  of 
cognitive  task  analysis.  The  organization  of  such  a  workshop  was  recommended  by  RSG.27  in  its  first  deliverable 
(Schraagen,  Chipman,  Shute,  Annett,  Strub,  Sheppard,  Ruisseau  &  Graff,  1997).  The  goal  of  the  workshop  was  to 
assess  the  state  of  the  art  by  bringing  together  a  diverse,  yet  representative  sample  of  experts  in  the  field.  The  outcome 
of  the  workshop  will  be  an  edited  book  and  a  report  to  NATO. 

2  ORGANIZATION  OF  THE  WORKSHOP 

In  its  meeting  of  March,  1997,  RSG.27  selected  potential  speakers  for  the  workshop.  In  June,  a  first  mailing  went  out  to 
potential  speakers,  inviting  them  to  submit  an  abstract  before  August  1.  In  parallel,  a  notice  was  posted  on  the  Cognitive 
Science  Society  Bulletin  Board,  in  order  to  attract  more  potential  speakers.  In  the  beginning  of  August,  50  abstracts 
were  received.  Of  these,  23  were  selected  for  presentation.  The  23  participants  were  asked  to  submit  a  draft  chapter  by 
October  1.  Most  of  the  participants  fulfilled  this  requirement. 

3  FINANCIAL  STRUCTURE  OF  THE  WORKSHOP 

Although  tire  workshop  was  organized  as  part  of  the  activities  of  RSG.27,  NATO  itself  did  not  provide  substantial 
support  for  the  workshop  (as  is  common  for  these  activities).  Fortunately,  the  American  representative  in  RSG.27,  Dr. 
Chipman  of  the  Office  of  Naval  Research,  was  able  to  provide  sufficient  funding  to  allow  the  workshop  to  proceed. 
Travel  costs  (including  accommodation  and  meals)  for  all  speakers  were  reimbursed  by  ONR.  Members  of  RSG.27  had 
to  provide  their  own  funding.  As  ONR,  being  a  government  agency,  could  not  legally  pay  for  individual  speakers,  the 
entire  funding  was  carried  over  to  Wright  State  University  in  the  form  of  a  grant.  Dr.  Shalin  of  Wright  State  University 
was  in  charge  of  the  grant  and  all  organizational  matters,  including  the  choice  of  venue  for  the  workshop  (the  Hay- 
Adams  Hotel).  The  executive  organizing  committee  consisted  of  the  Chairman  of  RSG.27,  Dr.  Schraagen,  and  the  US 
representative  in  the  RSG,  Dr.  Chipman.  The  executive  committee  determined  the  final  list  of  speakers,  assisted  by  Dr. 
Shalin. 

4  LIST  OF  SPEAKERS 

The  following  23  speakers  presented  their  paper  during  the  workshop  (country  of  citizenship  in  brackets): 

Barnard  (UK) 

Boy  (FR) 

Seamster  (USA) 

Vicente  (CAN) 
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Hoffman  (USA) 

Hamilton  (UK) 

Williams  (USA) 

DuBois  (USA) 

Schaafstal  (NETH) 

Gray  (USA) 

Flach  (USA) 

Hunt  (USA) 

Annett  (UK) 

Klein  (USA) 

Cooke  (USA) 

Zachary  (USA) 

Neerincx  (NETH) 

Paris  (AUS) 

Roth  (USA) 

Ormerod  (UK) 

Johnson  (UK) 

Woods  (USA) 

Kieras  (USA) 

In  total,  13  speakers  were  from  the  USA,  5  from  the  UK,  2  from  The  Netherlands,  1  from  France,  1  from  Canada,  and  1 
from  Australia. 

In  addition,  two  guest  speakers  gave  presentations.  The  first  was  Rear  Admiral  Huchting,  Program  Executive  Officer 
for  Surface  Combatants.  The  second  was  Dr.  Lesgold  of  the  Learning  Research  and  Development  Center,  University  of 
Pittsburgh. 

5  SUMMARY  OF  PRESENTATIONS 


All  presentations  were  grouped  into  6  sessions: 

Session  I:  General  frameworks  (Barnard,  Boy,  Seamster  &  Vicente) 

Session  II:  Tools  and  Techniques  (Hoffman,  Hamilton,  Williams,  DuBois) 

Session  III:  Facets  of  expertise  with  implications  for  CTA  (Schaafstal,  Gray,  Flach,  Hunt) 
Session  IV:  Team  CTA  (Annett,  Klein,  Cooke,  Zachary) 

Session  V:  CTA  for  design  I  (Neerincx,  Paris,  Roth,  Ormerod,  Johnson) 

Session  VI:  CTA  for  design  II  (Woods,  Kieras) 


Papers  were  either  30  or  40  minutes  in  length,  with  a  5  minute  question  and  answer  session  after  each  paper.  At  the  end 
of  each  session,  a  50  minute  general  discussion  session  was  planned.  A  discussant  was  assigned  to  each  session,  with 
the  role  of  initiating  and  leading  discussions.  The  discussant  for  Session  I  was  Schmalhofer  (GER),  for  Session  II: 
Shalin  (USA),  for  Session  III:  Hoffman  (USA),  for  Session  IV:  Schraagen  (NETH),  and  for  Session  V:  Vicente  (CAN). 


Session  I:  General  frameworks 


Barnard  (MRC  Applied  Psychology  Unit)  presented  the  Interacting  Cognitive  Subsystems  (ICS)  framework,  from 
which  a  form  of  CTA  of  broad  scope  and  applicability  has  been  derived.  For  any  task,  four  components  are  described: 

(1)  The  configuration  of  mental  processes  called  into  play 

(2)  Their  procedural  knowledge  or  processing  capability 

(3)  The  task  relevant  contents  of  memory  records  they  can  call  upon 

(4)  A  description  of  the  way  in  which  the  overall  mental  mechanism  is  controlled  and  coordinated. 

Applications  have  been  mainly  in  the  Human-Computer  Interaction  area.  The  focus  is  on  predictive  modeling  of  either 
existing  or  future  systems. 

Boy  (EURISCO)  distinguished  between  the  normative,  prescribed,  task  and  the  descriptive,  'effective',  task.  The 
cognitive  function  is  a  mapping  between  the  prescribed  task  and  the  effective  task.  The  cognitive  function  has  a  role,  a 
context,  and  resources  (user,  task,  artifact,  environment).  Transferring  cognitive  functions  from  humans  to  machines  is 
called  automation.  Cognitive  function  analysis  (CFA)  has  been  applied  mainly  in  the  domain  of  advanced  cockpit 
automation.  CFA  is  supported  by  a  cooperative  use  of  active  design  documents  (ADDs).  ADDs  are  shareable  prototypes 
of  the  real  artifacts  being  designed  that  can  be  used  by  real  users  to  assess  their  usability.  Users  work  from  the 
prescribed  task  directly  with  a  simulated  task.  In  this  way,  CTA  is  part  of  the  system  development,  not  an  activity  that  is 
done  up-front  first  and  the  results  of  which  are  then  handed  over  to  designers.  CFA  is  a  form  of  event-oriented  task 
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analysis,  in  contrast  to  goal-oriented  task  analysis.  It  is  suitable  for  complex,  open  system  environments  where  problem 
formulation  is  more  important  than  problem  solving. 


Seamster  (Cognitive  &  Human  Factors),  Redding,  and  Kaempf  have  developed  a  skill-based  CTA  framework  for 
practitioners  in  aviation.  The  framework  highlights  the  skill  or  performance  side  of  cognition  and  proposes  a  set  of 
complete  methods  to  analyze  cognitively  complex  tasks.  The  main  skill  types  in  the  framework,  together  with  relevant 
methods  for  analyzing  these  skill  types,  are: 

(1)  strategies:  metacognitive  skills  such  as  coordination  and  monitoring.  Relevant  methods:  Team  Communication 
Method  and  Verbal  Report  Method. 

(2)  decision  making  skills:  decision  making  mles  of  thumb.  Relevant  methods:  Critical  Decision  Method  and  Error 
Analysis  Method. 

(3)  representational  skills:  'mental  models'  that  allow  mental  simulation  and  the  prediction  of  outcomes.  Relevant 
methods:  Diagramming  Method  and  Rating/Sorting  Methods. 

(4)  procedural  skills:  building  blocks  of  skilled  performance.  Relevant  methods:  Simplified  PARI  Method  and 
Verbal  Report  Method. 

(5)  automated  skills:  foundation  of  skilled  performance.  Relevant  methods:  Consistent  Component  Method  and  Job 
Performance  Method. 

Vicente  (University  of  Toronto)  presented  as  his  main  aim  to  support  operators  in  turbulent  environments.  Most 
disturbances  in  these  environments  were  unanticipated  by  designers.  For  open  systems,  with  disturbances  that  need  to 
be  taken  into  account,  a  constraint-based  approach  to  task  analysis  is  more  appropriate  than  an  instruction-based 
approach.  A  constraint-based  approach  just  lists  constraints,  does  not  tell  you  the  right  way  to  do  your  task.  This  leads 
to  the  distinction  between  task  analysis  and  work  analysis:  in  task  analysis,  the  focus  is  on  efficiency  of  task 
performance,  while  in  work  analysis,  the  focus  is  on  flexibility  and  broad  scope  of  applicability.  An  analysis  of  the 
work  domain  (system)  should  precede  a  task  analysis,  at  least  in  open  systems.  Tools  that  may  be  used  in  a  work 
domain  analysis  are  the  decision  ladder  and  the  use  of  abstraction  hierarchies.  This  leads  to  a  functional  description  of 
systems. 

Session  II:  Tools  and  techniques 

Hoffman  (Adelphi  University)  presented  research  on  the  validity  and  reliability  of  one  CTA  method:  the  Critical 
Decision  Method  (CDM).  The  CDM  is  a  case-specific  multi-trial  retrospection,  structured  by  probe  questions,  designed 
to  elicit  information  about  the  important  cues,  choice  points,  options,  and  action  plans.  It  has  been  applied  in  areas  as 
diverse  as  neonatal  intensive  care,  fire  fighting,  and  anti-air  warfare.  Validity  and  reliability  of  the  CDM  have  generally 
been  high.  For  instance,  in  clinical  nursing  skill,  cues  derived  from  the  CDM  data  were  both  more  specific  and  more 
pertinent  than  the  cues  culled  from  the  literature.  Retest  reliability  and  inter-coder  agreement  have  been  in  the  80-90% 
range.  The  CDM  research  was  used  to  illustrate  how  a  number  of  CTA  methods  can  be  combined. 

Hamilton  (Human  Engineering  Ltd)  presented  a  software  tool  for  CTA  purposes,  called  ATLAS.  ATLAS  has  a 
GOMS-like  hierarchical  goal  structure.  For  specific  goals,  only  the  knowledge  that  is  relevant  for  achieving  that  goal  is 
specified.  Based  on  a  transaction  analysis,  the  following  elements  result:  goal,  initiation,  input,  knowledge  elements, 
transactions,  translation,  and  outcomes.  ATLAS  can  be  used  for  performance  time  predictions,  workload  analysis,  and 
reliability  analysis.  It  has  been  applied  to  more  than  15  projects. 

Williams  (Teknowledge  Corporation)  described  a  tool  for  lay  people  that  yields  total  task  execution  time  for  a  serial  list 
of  activities.  Parallel  activities  are  excluded.  The  focus  of  the  tool  is  on  production  mles,  not  declarative  memory.  The 
tool  is  therefore  primarily  applicable  to  routine  Human-Computer  Interaction  performance.  It  guides  the  user  through  a 
series  of  detailed  questions  at  the  key-stroke  level.  At  the  same  time,  questions  concerning  the  number  and  content  of 
chunks  have  to  be  answered. 

DuBois  (Psychological  Systems  and  Research,  Inc.)  focused  on  test  design  for  areas  such  as  land  navigation,  crew 
coordination,  and  computer  maintenance.  Four  phases  in  each  CTA  activity  were  distinguished: 

(1)  planning 

(2)  knowledge  elicitation  (mostly  protocol  analysis) 

(3)  knowledge  representation 

(4)  products  (error  lists,  task  &  method  lists,  detailed  test  specs) 

The  focus  was  on  the  domain  content  of  the  knowledge  representation.  Special  emphasis  was  given  to  multiple, 
conflicting  constraints  and  goals,  choice  in  response,  multiple  levels  of  abstraction. 

Session  III:  Facets  of  expertise  with  implications  for  CTA 
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Schaafstal  (TNO  Human  Factors  Research  Institute)  presented  the  results  of  a  successful  project  on  training  for 
troubleshooting,  carried  out  for  the  Royal  Netherlands  Navy.  In  the  project,  the  results  of  a  cognitive  task  analysis  of 
experienced  maintenance  technicians  were  used  to  develop  a  new  training  course  in  troubleshooting.  The  main  features 
of  the  new  course,  as  compared  with  the  regular  courses,  are:  more  emphasis  on  functional  system  understanding  and 
more  emphasis  on  a  systematic  approach  to  troubleshooting.  Initially,  the  new  course  was  a  one-day  add-on  to  an 
existing  course.  An  empirical  evaluation  showed  a  large  improvement  in  troubleshooting  performance  and  system 
understanding,  compared  with  the  regular  course.  As  these  results  could  have  been  due  to  the  lengthening  of  the  course 
rather  than  the  novel  contents,  a  completely  new  course  was  developed,  which  in  fact  resulted  in  a  4-week  instead  of  a 
6- week  course.  Even  this  reduced  course  resulted  in  better  performance  and  system  understanding  than  the  regular 
course.  The  Dutch  Navy  has  accepted  the  new  way  of  designing  courses,  based  on  the  cognitive  task  analysis,  as  the 
basis  for  all  courses  in  troubleshooting. 

Gray  (George  Mason  University)  discussed  some  lessons  learned  from  project  Nemo,  an  attempt  to  model  the  expertise 
of  the  submarine  Approach  Officer.  The  most  important  lesson  was  that  'needless  naturalism'  should  be  avoided,  that  is, 
that  high-fidelity  simulations  carry  with  it  the  risk  of  loss  of  experimental  control,  and  hence  lead  to  problems  in 
interpreting  the  data.  Instead,  'manageably  complex  systems'  should  be  used  for  research  purposes.  These  systems  are 
based  upon  complex  real-world  environments  and  extract  the  essence  of  those  environments  in  such  a  way  as  to  enable 
key  features  of  this  complexity  to  be  studied. 

Flach  (Wright  State  University)  emphasized  the  importance  of  studying  the  work  domain  of  experts,  as  the  constraints 
put  upon  the  experts  by  the  environment  they  operate  in,  determine,  to  a  large  extent,  their  behavior.  This  general 
principle  then  leads  to  a  variety  of  methods  for  cognitive  task  analysis:  table  top  analysis  (read  books  before  talking  to 
experts),  knowledge  elicitation  (across  a  wide  sample  of  perspectives  and  experts:  include  operators,  designers, 
managers,  scientists),  naturalistic  observation  (watch  real  phenomena;  try  to  perform  the  task  yourself),  laboratory 
studies  (necessarily  reductionistic;  use  stimuli  that  are  meaningful  with  respect  to  the  phenomenon  you  are  interested 
in),  synthetic  task  environments  (high-fidelity  simulation). 

Hunt  (University  of  Washington)  presented  results  of  an  empirical  study  on  public  safety  dispatching.  Based  on  on-site 
observations  of  dispatchers  (and  not  a  formal  CTA),  Hunt  and  his  co-workers  were  struck  by  the  differences  in  skills  of 
dispatchers.  They  formed  the  hypothesis  that  the  basis  for  these  skill  differences  lay  in  the  ability  to  handle  different 
kinds  of  events  at  the  same  time.  The  question  arose  whether  this  was  an  abstract  task  or  a  highly  context-specific  one. 
An  abstract  decision-making  task  was  developed  which  supposedly  captured  the  essence  of  the  dispatching  task.  The 
real  dispatching  task  was  also  simulated,  and  correlations  in  performance  were  determined  for  both  tasks.  It  turned  out 
that  well-trained  students,  performing  as  well  as  dispatchers  on  a  dispatching  simulation  task,  showed  a  correlation  of 
.70  between  the  abstract  decision  making  task  and  the  dispatching  simulation.  The  correlation  was  .50  with  real 
dispatchers.  The  importance  of  this  work  is  that  observations  were  submitted  to  an  empirical  test.  What  was  not 
captured  in  the  simulations  was  the  ability  to  do  really  uncommon  problem  solving,  e.g.,  talking  someone  out  of 
committing  suicide  or  combining  two  different  views  of  the  same  incident. 

Session  IV:  Team  CTA 

Annett  (University  of  Warwick)  described  a  study  in  which  Hierarchical  Task  Analysis  was  applied  to  the  team  level, 
in  this  case  a  naval  command  team.  The  purpose  of  the  study  was  to  improve  team  training.  As  a  first  step,  a 
scientifically-based  objective  measurement  of  team  performance  was  carried  out.  This  measurement  was  guided  by  a 
model  of  teamwork  that  distinguishes  between  team  product  and  team  process.  As  the  recording  of  communication  in  a 
command  team  was  too  expensive,  questionnaires  were  used  instead  for  measuring  team  process  variables.  As  team 
product  variables  were  taken  the  times  at  which  the  teams  carried  out  certain  critical  behaviors.  The  results  showed  no 
relationship  between  process  and  product.  Instead,  technical  competence  of  individual  team  members  accounted  for 
most  of  the  variance  in  team  performance. 

Klein  (Klein  Associates)  identified  a  small  set  of  cognitive  processes  for  teams  that  guide  the  analysis  of  team  CTA. 
The  5  key  processes  are:  control  of  attention  (information  management),  shared  situation  awareness,  shared  mental 
models,  application  of  strategies  and  heuristics  to  make  decisions,  metacognition.  A  further  distinction  can  be  made 
between  planning  teams  and  action  teams.  The  job  of  a  planning  team  is  to  produce  a  plan  (e.g.,  command  and  control 
teams).  The  job  of  an  action  team  is  to  accomplish  a  task  (e.g.,  carrying  out  a  plan  created  by  higher  echelons).  Results 
of  two  team  CTA  projects  (one  for  the  U.S.  Marine  Corps  command  posts  and  the  other  for  an  emergency  response 
organization  of  a  nuclear  power  plant)  showed  that  team  members  are  often  confused  about  each  others'  roles.  This 
results  in  information  overload  (in  accordance  with  the  maxim  "when  in  doubt,  send  it  out"),  a  sign  of  ineffective 
information  management.  The  output  of  the  team  CTA  needs  to  describe  the  5  key  processes.  The  techniques  that  are 
often  used  are  observations,  interviews  and  simulations. 
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Cooke  (New  Mexico  State  University)  focused  on  team  cognition  as  an  emerging  feature  of  the  interplay  of  the 
cognition  of  individual  team  members.  One  important  aspect  of  team  cognition  is  shared  knowledge.  Three  types  of 
sharing  can  be  distinguished:  common,  complementary  and  conflicting.  An  important  question  in  this  respect  is  what 
knowledge  should  be  shared  by  teammates  for  effective  task  performance  and  how  is  this  knowledge  most  effectively 
shared.  Cooke  then  focussed  on  an  agenda  for  Cognitive  Team-Task  Analysis.  First,  knowledge  of  individuals  should 
be  measured  and  compared  to  a  referent  (using  similarity  metrics)  to  assess  accuracy.  Second,  each  individual's 
knowledge  should  be  compared  to  other  team  members'  knowledge  (also  using  similarity  metrics).  Third,  the  question: 
Are  patterns  of  common,  complementary  or  conflicting  knowledge  predictive  of  performance?,  may  be  answered. 

Zachary  (CHI  Systems,  Inc.)  introduced  ORGNET,  a  novel  language  for  describing  cognitive  processes  at  the  team 
level.  ORGNET  is  abstracted  upwards  from  an  individual-level  cognitive  modeling  language  called  COGNET. 
COGNET  should  be  supplemented  by  a  set  of  principles  for  allocating  cognitive  functions  among  a  team  of  people 
based  on  existing  literature.  One  principle,  for  instance,  is  to  minimize  required  communication  among  team  members. 
The  question  is  whether  these  principles  are  general  enough  to  be  applied  to  all  kinds  of  teams  (e.g.,  for  some  well¬ 
performing  teams,  it  has  been  found  that  they  are  characterized  by  intense  communication). 

Session  V :  CTA  for  design 

Neerincx  (TNO  Human  Factors  Research  Institute)  presented  an  integrated  method  for  Cognitive  Task  Analysis, 
including  cognitive  task  load  analysis  and  a  design  method  for  user  interfaces  providing  cognitive  support  in  high  task 
load  situations.  The  method  was  applied  to  the  user  interface  development  of  a  future  Ship  Control  Centre.  The  outcome 
of  the  CTA  is  used  for  software  development  for  platform  control  systems.  The  CTA  consists  of  a  task  break-down, 
event  descriptions,  task  allocation  to  persons,  scenario  narratives  and  action  sequence  charts.  In  this  way,  it  is  possible 
to  predict,  for  each  scenario,  and  for  each  person,  the  percentage  knowledge-based  actions,  the  number  of  task-set 
switches,  and  the  percentage  of  time  occupied.  Together,  these  results  indicate  moments  of  high  peak  load.  These 
results  will  be  used  in  the  design  phase  to  improve  the  proposed  user  interface,  in  order  to  reduce  peak  load. 

Paris  (CSIRO  Mathematical  and  Information  Sciences)  described  two  case  studies  in  which  task  models  were  used  to 
bridge  the  gap  between  software  engineering  and  Human-Computer  Interaction.  The  first  case  study  concerned  the 
design  of  a  decision-support  software  system  for  the  Australian  Navy.  The  problem  from  an  analysis  point  of  view  was 
the  absence  of  a  common  model  among  the  users.  The  task  model  used  (MAD)  enabled  communication  among  the 
software  designers,  the  end-users,  the  interface  designers  and  the  technical  writers.  Structured  interviews  were  used  to 
provide  the  knowledge  and  representation.  The  new  system  has  reduced  training  time  from  6  months  to  a  few  weeks 
and  performance  has  improved.  The  second  case  study,  Isolde,  involves  the  development  of  an  authoring  tool  for  on¬ 
line  documentation.  This  tool  is  developed  for  technical  writers  who  have  to  write  on-line  help  after  the  software  has 
been  developed.  Their  task  was  studied  first.  It  turned  out  that  technical  writers  do  not  understand  the  software  first,  as 
is  prescribed,  but  develop  a  mental  model  based  on  a  partial  understanding  and  start  writing  immediately.  In  this  second 
case  study,  the  task  models  are  used  to  represent  the  functionality  of  the  software  from  which  documentation  is 
automatically  generated.  Task  models  are  also  used  to  represent  the  cognitive  task  of  technical  writing. 

Roth  (Roth  Cognitive  Engineering)  emphasized  that  CTA  is  more  than  the  application  of  any  single  technique.  CTA  is 
a  modeling  effort,  aimed  at  modeling  both  the  domain  and  the  expertise  and  error.  First,  the  analyst  needs  to  understand 
the  way  the  world  works  and  what  makes  the  world  hard  (e.g.,  keeping  track  of  events  is  difficult  because  the  world  is 
constantly  changing).  Current  methods  and  tools  for  this  effort  are:  functional  means-ends  analysis,  structured 
interview  methods,  functional  task  modeling,  ethnographic-observational  investigations.  Second,  the  analyst  needs  to 
understand  the  way  people  operate  in  their  world  (to  model  their  expertise,  knowledge,  strategies,  error).  Current 
techniques  are:  semantic  mapping,  critical  incident  technique,  critical  decision  method,  ethnographic-observational 
investigations,  and  structured  interview  methods.  After  that,  the  envisioned  world  needs  to  be  explored:  how  will 
envisioned  technological  change  shape  cognition  and  collaboration  and  how  will  practitioners  adapt  artifacts  to  meet 
their  own  goals?  To  answer  this  question,  the  analyst  can  again  focus  on  the  domain  and  the  practitioner.  Typical 
techniques  for  the  former  are  scenario  generation  and  for  the  latter  storyboard  walkthroughs,  participatory  design, 
wizard-of-Oz  technique  and  high-fidelity  simulations. 

Ormerod  (Lancaster  University)  discussed  the  role  of  CTA  in  the  design  process.  He  noted  that  the  design  community 
has  been  slow  in  adopting  task  analytic  methods.  This  is  partly  due  to  the  device  dependence  of  most  methods,  and  the 
failure  to  make  explicit  the  stopping  conditions  for  task  analysis.  Moreover,  most  methods  are  difficult  to  leam  by  non¬ 
specialists.  The  sub-goal  template  methodology  (SGT)  was  proposed  to  remedy  these  problems.  SGT  starts  with  a 
Hierarchical  Task  Analysis,  but  adds  sub-goal  templates  or  standard  task  elements.  Four  classes  of  elements  are 
distinguished:  action,  communication,  monitoring  and  decision  making  elements.  Each  of  these  can  be  decomposed 
further.  Next,  sequencing  elements  are  added  to  specify  the  way  information  should  be  arranged  for  presentation  to  the 
user.  The  SGT  method  has  been  applied  primarily  to  interface  design  tasks.  A  computer-based  tool  has  been  developed 
to  support  its  use.  As  yet,  no  evaluation  studies  of  the  SGT  method  and  tool  have  been  carried  out. 
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Johnson  (University  of  London)  developed  Task  Knowledge  Structures  (TKS)  for  the  purpose  of  user  interface  design. 
Starting  with  an  existing  task  model,  an  abstraction  process  yields  an  envisioned  task  model,  which  in  turn  results  in  the 
design  of  new  interfaces.  The  abstraction  process  is  guided  by  principles  of  similarity,  causality,  conformance  and 
transformation.  For  instance,  the  principle  of  similarity  states  that  objects  that  are  the  same  or  similar  will  be 
conceptually  grouped  together,  and  actions  on  the  same  or  similar  objects  will  be  carried  out  together.  Empirical  tests  of 
these  principles  showed  that  interfaces  developed  according  to  these  principles  led  to  fewer  errors  and  faster 
performance.  In  the  case  of  unprincipled  design,  users  will  impose  their  own  task  structure  but  this  will  take  longer  to 
achieve.  The  use  of  these  design  principles  puts  task  analysis  in  front  of  the  design  process,  in  a  prescriptive  instead  of 
evaluative  role. 

Woods  (The  Ohio  State  University)  sketched  the  envisioned  world  problem:  there  are  many  different  views  of  the 
future  world  (problem  of  plurality)  and  these  views  are  under-specified.  The  challenge  for  CTA  is  to  deal  with  this 
problem  in  the  design  of  new  systems,  since  the  introduction  of  new  technology  will  transform  the  nature  of  practice. 
This,  in  turn,  means  that  the  nature  of  the  cognitive  activities  will  change,  hence,  CTA  should  anticipate  this  change. 
Effective  CTA  must  face  a  challenge  of  prediction.  As  an  example,  the  proposed  Free  flight  system  was  discussed  and 
the  issue  was  to  predict  accidents  that  had  not  yet  occurred.  It  turned  out  that  scenario  design  (or  the  future  incident 
method)  was  a  valuable  tool:  different  kinds  of  participants  in  the  current  system  view  Ml  incident  reports  about  a 
possible  future  world.  It  was  important  to  incorporate  different  perspectives  (controllers,  pilots,  dispatchers),  in  order  to 
deal  with  the  problems  of  plurality  and  under-specification. 

Kieras  (University  of  Michigan)  emphasized  the  importance  of  task  strategies  over  the  architecture  or  the 
representation  of  task-specific  inputs.  Particularly  in  complex  militarily  relevant  tasks,  performance  depends  very 
heavily  on  the  task  strategy.  The  problem  is:  what  strategies  are  people  going  to  use  in  novel  systems?  Often,  the 
strategy  is  not  obvious,  even  to  "experts".  This  makes  system  design  with  predictive  models  of  human  performance  very 
difficult.  Kieras  described  the  "bracketing"  heuristic  to  address  this  issue.  Starting  with  a  plausible  cognitive 
architecture  (e.g.,  EPIC),  one  can  define  a  "fastest-possible"  and  a  "slowest-reasonable"  strategy.  The  former  strategy 
matches  human  performance  when  humans  place  a  high  priority  on  speed  and  use  parallel  processing;  the  latter  strategy 
is  best  characterized  as  deliberate  and  unhurried.  Actual  human  performance  should  lie  between  these  two  extreme 
strategies,  and  this  is  confirmed  by  experimental  data.  Devising  the  two  bracketing  strategies  requires  a  task  analysis 
that  specifies  what  people  should  do,  not  what  they  actually  do.  Existing  CTA  methods  are  satisfactory  for  this  purpose. 

Guest  speakers 

Huchting  (Program  Executive  Office  Surface  Combatants/ Aegis  Program)  described  the  recent  changes  the  Navy  is 
faced  with.  The  environments  are  much  more  demanding  these  days.  Failure  in  decision  making  is  not  allowed  any 
more.  The  difference  with  civilian  environments  is  that  "late  is  wrong".  The  use  of  commercial  products  will  increase: 
after  the  year  2000,  the  US  Navy  will  buy  Commercial  Off  the  Shelf  products  only,  no  standard  Navy  computer 
equipment  any  more.  The  challenge  cognitive  psychologists  face  is  to  get  in  front  of  the  manufacturer,  and  help  them 
design  systems  so  that  they  are  usable.  The  sailors  on  board  are  faced  with  the  challenge  to  use  all  the  new  technology. 
Embedded  training  is  very  important  in  this  respect,  more  so  than  display  design.  In  the  Vincennes  incident,  it  turned 
out  that  humans  were  not  well  trained.  The  Aegis  system  has  an  automatic  data  recording  system  and  later  analysis  of 
the  system  showed  that  the  system  knew  that  the  Iranian  Airbus  was  rising  (and  hence  had  no  hostile  intentions). 
Apparently,  the  human  operators  did  not  get  this  information  out  of  the  system,  or,  if  they  did,  they  remembered  things 
differently  than  they  really  did,  because  the  operators  said  the  Airbus  was  descending  towards  them. 

Lesgold  (University  of  Pittsburgh)  gave  an  overall  perspective  on  the  papers  presented  during  the  workshop.  He  started 
by  briefly  summarizing  the  purposes  of  CTA:  design  of  interfaces,  design  of  work  procedures  and  design  of  training.  He 
noted  that  there  are  good  techniques  for  designing  interfaces,  at  least  those  for  routine  tasks.  For  non-routine  tasks, 
schemes  for  opening  up  the  analysis  are  evolving.  The  challenges  for  CTA  lie  in  the  areas  of  team  work,  task  selection 
(do  people  really  do  what  we  think  they  do?),  and  high  rates  of  turnover  in  personnel  (how  to  fit  in  a  new  team  member 
into  an  existing  team?).  CTA  should  focus  more  on  cases  where  systems  and  knowledge  do  not  fit  very  well,  because 
once  you  really  understand  the  task,  it  is  a  good  candidate  for  automation  and  there  is  no  need  for  CTA  any  more.  Even 
though  the  scale  of  automation  keeps  increasing,  there  remains  a  need  for  operator  flexibility  and  adaptation.  Instead  of 
focusing  on  the  task  as  performed,  or  the  work  as  observed,  analysts  should  focus  more  on  the  negotiation  of  the 
problem  character  or  the  problem  solution  requirements.  Finally,  cognitive  task  analysts  should  take  a  look  at  object- 
oriented  design  and  see  what  they  can  leam  from  the  software  engineers. 

6  GENERAL  ISSUES 

Design  of  new  systems 

One  issue  that  dominated  the  workshop  was  the  issue  of  the  use  of  CTA  in  the  design  of  new  systems.  Many  speakers 
addressed  this  issue  and  different  tentative  solutions  were  proposed.  The  difficulty  lies  in  the  fact  that  when  we  try  to 
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predict  future  operator  behavior,  we  can  only  rely  on  existing  operator  behavior.  As  analysts,  we  therefore  either  have  to 
make  informed  guesses  about  the  strategies  that  operators  are  going  to  use,  or  we  have  to  elicit  current  expert  opinion 
with  regard  to  future  expert  behavior.  Using  a  wide  range  of  scenarios  of  possible  future  worlds  and  incorporating  a 
wide  range  of  perspectives  of  current  experts  are  prerequisites  for  successfully  addressing  the  "envisioned  world 
problem".  Of  course,  the  analyst  may  always  be  charged  with  taking  too  few  scenarios  into  account;  the  list  of  possible 
future  events  can  be  extended  infinitely.  The  only  way  an  analyst  can  respond  to  this  charge,  is  to  use  principles  in  the 
selection  of  scenarios,  thereby  giving  some  indication  of  the  amount  of  coverage  of  future  events. 

Open  versus  closed  domains 

Another  dominant  theme  was  the  different  emphasis  put  by  different  speakers  on  either  the  cognitive  architecture  or  the 
analysis  of  the  work  domain.  The  emphasis  on  cognitive  modeling  and  cognitive  architectures  is  appropriate  for 
relatively  closed  and  static  domains,  but  is  inappropriate,  according  to  some  speakers,  in  relatively  open  and  dynamic 
domains.  In  open  domains,  the  analysis  should  start  with  a  functional  description  of  the  system  constraints,  before  going 
into  an  analysis  of  (routine)  tasks.  It  was  pointed  out  by  a  number  of  speakers  that  multiple  conflicting  goals  will  remain 
in  the  future  a  major  source  of  cognitive  difficulty.  Number  of  rules,  reaction  times,  and  system  interaction  will  become 
less  important  as  automation  proceeds.  Due  to  automation,  more  has  to  be  done  at  the  level  of  minutes  instead  of 
seconds.  This  would  imply  a  shift  in  emphasis  from  'task  analysis'  to  'work  analysis',  and  a  less  dominant  role  for 
cognitive  architectures  that  try  to  predict  user  behavior  at  the  level  of  seconds  or  below. 

Selection  of  appropriate  techniques 

The  question  when  to  use  what  technique  was  touched  upon  in  various  presentations  and  in  the  general  discussion 
sessions.  It  was  generally  felt  that  a  strict  and  fixed  ordering  of  techniques  for  each  project  was  unrealistic,  as  the  order 
in  which  techniques  are  used  may  differ  from  project  to  project.  Also,  use  of  a  single  technique  was  generally  felt  to  be 
too  restrictive:  multiple,  coordinated  approaches  to  CTA  are  required.  This  is  not  to  say  that  no  guidelines  at  all  can  be 
given  for  practitioners.  A  difficult  matter  remains  according  to  what  dimensions  techniques  should  be  classified.  Some 
suggested  a  coupling  of  technique  to  type  of  knowledge  to  be  elicited,  some  suggested  that  different  techniques  are 
needed  for  open-ended  tasks  than  for  closed  tasks,  and  some  suggested  that  different  techniques  are  needed  for 
understanding  either  the  domain  or  the  practitioners. 

Role  of  CTA  in  system  design 

Most  speakers  agreed  that  CTA  should  not  be  a  self-contained  activity,  the  results  of  which  are  handed  off  to  system 
designers.  However,  this  being  said,  it  is  still  very  often  the  case  that  there  is  no  integration  between  the  activities  of 
cognitive  psychologists  and  system  designers  (e.g.,  software  engineers).  It  is  probably  still  unclear  what  designers  need 
in  the  first  place,  and  the  CTA  community  needs  to  explicitly  reflect  on  this.  Some  authors  addressed  this  issue  and 
developed  methods  that  would  be  of  more  use  to  software  designers.  An  approach  that  has  not  been  taken  so  often,  is  to 
look  at  current  software  engineering  methods  and  to  try  to  incorporate  cognitive  task  analysis  notions  into  those 
methods.  An  example  of  successful  integration  of  CTA  techniques  in  the  design  process  are  the  "active  design 
documents".  At  any  rate,  it  is  critical  for  the  success  of  CTA  to  be  involved  in  the  design  process  from  the  start  to  finish, 
and  to  establish  clear  links  with  methods  that  are  used  by  other  disciplines. 

Value  of  CTA 

It  is  important  for  the  CTA  community  to  be  able  to  empirically  demonstrate  the  added  value  of  a  CTA.  In  this  way,  the 
analyst  goes  beyond  mere  observations  and  submits  his  or  her  ideas  to  empirical  tests.  Clear  performance  factors  need 
to  be  chosen  and  to  be  engineered  toward.  Examples  of  performance  factors  are  reaction  time,  training  time,  faults 
detected,  firepower,  coverage  of  weapon  systems,  etc.  Projects  in  which  the  effectiveness  of  CTA  was  measured 
included  the  redesign  of  training  courses  for  troubleshooting,  which  resulted  in  a  100%  performance  improvement  and  a 
reduction  in  training  time  by  10%  to  50%,  and  the  design  of  a  decision  support  system,  which  reduced  training  time 
from  6  months  to  a  few  weeks  and  improved  performance. 

Assessment  of  reliability  and  validity  of  CTA  techniques 

Due  to  the  fact  that  customers  are  rarely  willing  to  pay  for  an  extra  CTA,  or  the  same  CTA  carried  out  by  a  different 
analyst,  empirical  tests  of  the  reliability  of  CTA  techniques  are  rare.  A  notable  example  concerns  the  Critical  Decision 
Method  that  has  been  found  to  be  sufficiently  reliable  and  valid. 

Team  CTA 

An  important  new  theme  that  emerged  from  the  workshop  concerns  the  application  of  CTA  techniques  to  team  tasks.  It 
is  perhaps  odd  that  CTA  has  always  limited  itself  to  individuals,  since  cognition  in  real-life  settings  is  always 
distributed  across  humans  and  machines.  It  appeared  that  techniques  that  were  originally  designed  for  individual  level 
tasks  could  be  upscaled  to  team  level  tasks  (HTA,  scaling  techniques,  critical  decision  method,  COGNET).  This 
notwithstanding,  the  result  from  a  large  number  of  studies  indicate  that  "a  team  of  experts  is  not  an  expert  team",  in 
other  words,  that  there  are  specific  team  skills  that  determine  team  effectiveness  over  and  above  individual  proficiency. 
Important  research  questions  concern  the  nature  of  shared  mental  models  (what  exactly  needs  to  be  shared?),  and  of 
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team  situation  awareness  (is  it  more  than  the  sum  of  individual  SA's?).  A  number  of  broader  issues  was  raised  regarding 
teams:  the  role  of  personality  (team  roles),  distributed  decision  making,  social  skills,  leadership  skills,  morale,  cohesion, 
and  turnover  in  teams.  These  issues  do  not  seem  to  have  the  highest  priority  on  the  research  agenda  of  cognitive 
psychologists. 

Some  issues  that  were  not  discussed 

•  CTA  for  tasks  that  are  difficult  to  verbalize  (e.g.,  spatial  tasks 

•  Individual  differences  (culture,  nationality)  and  their  implication  for  solutions 

•  Effects  of  environmental  stressors  on  the  conduct  of  CTA 

•  Selecting/defining  expertise 
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